Max level shown:
12025-05-08T22:00:01.934ZINFOnexus (dropshot): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:50702 remote_addr = [fd00:1122:3344:101::d]:51203
22025-05-08T22:00:01.945ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 11588 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 8020d3d4-1462-40ee-ba6b-09da21a93d90 response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
32025-05-08T22:00:08.263ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 4 reason = Timeout
42025-05-08T22:00:08.263ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
52025-05-08T22:00:08.263ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 4 reason = Timeout
62025-05-08T22:00:08.263ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
72025-05-08T22:00:08.264ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
82025-05-08T22:00:08.264ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
92025-05-08T22:00:08.266ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 10 reason = Timeout
102025-05-08T22:00:08.266ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 4 reason = Timeout
112025-05-08T22:00:08.266ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
122025-05-08T22:00:08.266ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 984, tv_nsec: 9670522 } })
132025-05-08T22:00:08.266ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
142025-05-08T22:00:08.266ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163
152025-05-08T22:00:08.266ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 984, tv_nsec: 10157747 } })
162025-05-08T22:00:08.267ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1163
172025-05-08T22:00:08.267ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
182025-05-08T22:00:08.267ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 4 reason = Timeout
192025-05-08T22:00:08.267ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
202025-05-08T22:00:08.268ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
212025-05-08T22:00:08.270ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
222025-05-08T22:00:08.270ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
232025-05-08T22:00:08.270ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
242025-05-08T22:00:08.271ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
252025-05-08T22:00:08.271ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 5.236836ms iteration = 4 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
262025-05-08T22:00:08.272ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
272025-05-08T22:00:08.272ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
282025-05-08T22:00:08.273ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
292025-05-08T22:00:08.273ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
302025-05-08T22:00:08.273ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
312025-05-08T22:00:08.274ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
322025-05-08T22:00:08.277ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
332025-05-08T22:00:08.277ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
342025-05-08T22:00:08.277ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
352025-05-08T22:00:08.278ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 4 reason = Timeout
362025-05-08T22:00:08.278ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 5 reason = Timeout
372025-05-08T22:00:08.278ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
382025-05-08T22:00:08.278ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
392025-05-08T22:00:08.278ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 4 reason = Timeout
402025-05-08T22:00:08.278ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
412025-05-08T22:00:08.278ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
422025-05-08T22:00:08.278ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 4 reason = Timeout
432025-05-08T22:00:08.278ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 4 reason = Timeout
442025-05-08T22:00:08.278ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
452025-05-08T22:00:08.278ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
462025-05-08T22:00:08.278ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 4 reason = Timeout
472025-05-08T22:00:08.278ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
482025-05-08T22:00:08.278ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 4 reason = Timeout
492025-05-08T22:00:08.278ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
502025-05-08T22:00:08.278ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 4 reason = Timeout
512025-05-08T22:00:08.278ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
522025-05-08T22:00:08.278ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
532025-05-08T22:00:08.278ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 10 reason = Timeout
542025-05-08T22:00:08.278ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
552025-05-08T22:00:08.279ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
562025-05-08T22:00:08.279ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
572025-05-08T22:00:08.279ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
582025-05-08T22:00:08.279ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
592025-05-08T22:00:08.279ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
602025-05-08T22:00:08.279ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
612025-05-08T22:00:08.279ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
622025-05-08T22:00:08.279ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
632025-05-08T22:00:08.279ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
642025-05-08T22:00:08.280ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
652025-05-08T22:00:08.281ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 4 reason = Timeout
662025-05-08T22:00:08.281ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
672025-05-08T22:00:08.281ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 4 reason = Timeout
682025-05-08T22:00:08.281ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
692025-05-08T22:00:08.281ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 4 reason = Timeout
702025-05-08T22:00:08.281ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
712025-05-08T22:00:08.281ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 4 reason = Timeout
722025-05-08T22:00:08.281ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 4 reason = Timeout
732025-05-08T22:00:08.281ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
742025-05-08T22:00:08.281ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
752025-05-08T22:00:08.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
762025-05-08T22:00:08.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
772025-05-08T22:00:08.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
782025-05-08T22:00:08.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
792025-05-08T22:00:08.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
802025-05-08T22:00:08.283ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
812025-05-08T22:00:08.283ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
822025-05-08T22:00:08.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
832025-05-08T22:00:08.284ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
842025-05-08T22:00:08.284ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 18.67852ms iteration = 10 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
852025-05-08T22:00:08.295ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
862025-05-08T22:00:08.296ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
872025-05-08T22:00:08.347ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
882025-05-08T22:00:08.347ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
892025-05-08T22:00:08.348ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb76d817-dd3a-4cbf-985e-c9d2bd620673", "content-length": "26", "date": "Thu, 08 May 2025 22:00:08 GMT"} })
902025-05-08T22:00:08.348ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
912025-05-08T22:00:08.349ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
922025-05-08T22:00:08.350ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
932025-05-08T22:00:08.351ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
942025-05-08T22:00:08.354ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
952025-05-08T22:00:08.355ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
962025-05-08T22:00:08.356ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
972025-05-08T22:00:08.358ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
982025-05-08T22:00:08.358ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
992025-05-08T22:00:08.358ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
1002025-05-08T22:00:08.360ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
1012025-05-08T22:00:08.360ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 81.532412ms iteration = 4 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
1022025-05-08T22:00:08.360ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
1032025-05-08T22:00:08.360ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
1042025-05-08T22:00:08.360ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
1052025-05-08T22:00:08.360ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1062025-05-08T22:00:08.360ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
1072025-05-08T22:00:08.360ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
1082025-05-08T22:00:08.361ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1092025-05-08T22:00:08.362ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1102025-05-08T22:00:08.362ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
1112025-05-08T22:00:08.362ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
1122025-05-08T22:00:08.362ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0e182be-9ce7-4344-a44f-58860eef6fd9", "content-length": "26", "date": "Thu, 08 May 2025 22:00:08 GMT"} })
1132025-05-08T22:00:08.362ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
1142025-05-08T22:00:08.364ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
1152025-05-08T22:00:08.364ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
1162025-05-08T22:00:08.364ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
1172025-05-08T22:00:08.364ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
1182025-05-08T22:00:08.365ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 84.054109ms iteration = 4 status = {"errors":[],"garbage_collect_requested":[]}
1192025-05-08T22:00:08.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
1202025-05-08T22:00:08.365ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
1212025-05-08T22:00:08.366ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
1222025-05-08T22:00:08.366ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
1232025-05-08T22:00:08.366ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
1242025-05-08T22:00:08.366ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1252025-05-08T22:00:08.366ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
1262025-05-08T22:00:08.366ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1272025-05-08T22:00:08.368ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
1282025-05-08T22:00:08.368ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
1292025-05-08T22:00:08.369ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1302025-05-08T22:00:08.369ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
1312025-05-08T22:00:08.369ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
1322025-05-08T22:00:08.369ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
1332025-05-08T22:00:08.370ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
1342025-05-08T22:00:08.370ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1352025-05-08T22:00:08.370ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1362025-05-08T22:00:08.372ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
1372025-05-08T22:00:08.372ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
1382025-05-08T22:00:08.372ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
1392025-05-08T22:00:08.372ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 91.341366ms iteration = 4 status = {"errors":[],"finish_invoked_ok":[]}
1402025-05-08T22:00:08.372ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 91.852992ms iteration = 4 status = {"errors":[],"requests_created_ok":[]}
1412025-05-08T22:00:08.392ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1422025-05-08T22:00:08.393ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1432025-05-08T22:00:08.402ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1442025-05-08T22:00:08.402ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1452025-05-08T22:00:08.403ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1462025-05-08T22:00:08.408ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 2f2f2c32-8d58-4801-a669-9618eac0c3f7, lookup_type: ById(2f2f2c32-8d58-4801-a669-9618eac0c3f7) } result = Ok(())
1472025-05-08T22:00:08.408ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1482025-05-08T22:00:08.408ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1492025-05-08T22:00:08.409ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
1502025-05-08T22:00:08.409ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
1512025-05-08T22:00:08.409ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1522025-05-08T22:00:08.410ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 131.437156ms iteration = 5 status = {"physical_disks_added":0}
1532025-05-08T22:00:08.411ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 132.415576ms iteration = 4 status = 0
1542025-05-08T22:00:08.411ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 130.418265ms iteration = 4 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
1552025-05-08T22:00:08.413ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
1562025-05-08T22:00:08.413ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
1572025-05-08T22:00:08.413ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 149.522229ms iteration = 4 status = {}
1582025-05-08T22:00:08.413ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
1592025-05-08T22:00:08.415ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 136.088626ms iteration = 10 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
1602025-05-08T22:00:08.417ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:552 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
1612025-05-08T22:00:08.417ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 138.583232ms iteration = 4 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
1622025-05-08T22:00:08.418ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
1632025-05-08T22:00:08.418ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
1642025-05-08T22:00:08.418ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 139.756715ms iteration = 4 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
1652025-05-08T22:00:08.419ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1662025-05-08T22:00:08.421ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
1672025-05-08T22:00:08.421ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
1682025-05-08T22:00:08.421ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
1692025-05-08T22:00:08.424ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
1702025-05-08T22:00:08.424ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
1712025-05-08T22:00:08.426ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 163.160855ms iteration = 4 status = null
1722025-05-08T22:00:08.447ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 166.279618ms iteration = 4 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
1732025-05-08T22:00:08.452ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 984, tv_nsec: 196082266 } })
1742025-05-08T22:00:08.453ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
1752025-05-08T22:00:08.457ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:362
1762025-05-08T22:00:08.457ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 189.594918ms iteration = 4 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
1772025-05-08T22:00:08.465ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
1782025-05-08T22:00:08.465ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
1792025-05-08T22:00:08.475ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
1802025-05-08T22:00:08.475ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
1812025-05-08T22:00:08.477ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1822025-05-08T22:00:08.479ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
1832025-05-08T22:00:08.479ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
1842025-05-08T22:00:08.479ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
1852025-05-08T22:00:08.481ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1862025-05-08T22:00:08.482ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e zone_address = fd00:1122:3344:101::2
1872025-05-08T22:00:08.482ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
1882025-05-08T22:00:08.482ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
1892025-05-08T22:00:08.482ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
1902025-05-08T22:00:08.482ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "64ccc4ff-a499-4ef6-b1a1-18c1cd715059", "content-length": "26", "date": "Thu, 08 May 2025 22:00:08 GMT"} })
1912025-05-08T22:00:08.482ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1174 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
1922025-05-08T22:00:08.482ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
1932025-05-08T22:00:08.483ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
1942025-05-08T22:00:08.564ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
1952025-05-08T22:00:08.564ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "91bbfd90-87e2-4427-a74e-3a9ced0382fc", "content-length": "691", "date": "Thu, 08 May 2025 22:00:08 GMT"} })
1962025-05-08T22:00:08.565ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
1972025-05-08T22:00:08.565ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
1982025-05-08T22:00:08.565ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
1992025-05-08T22:00:08.566ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 287.429394ms iteration = 4 status = {}
2002025-05-08T22:00:08.601ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
2012025-05-08T22:00:08.601ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
2022025-05-08T22:00:08.602ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ee54745-f97b-4a88-8a66-97a1a2ebecf6", "content-length": "372", "date": "Thu, 08 May 2025 22:00:08 GMT"} })
2032025-05-08T22:00:08.602ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
2042025-05-08T22:00:08.602ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
2052025-05-08T22:00:08.645ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
2062025-05-08T22:00:08.646ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
2072025-05-08T22:00:08.649ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: cd0b930c-534b-46ad-ac3b-2eaa94b0bf2c, rack_id: 64f3d6f4-89b8-48aa-a622-b684e261667e, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(65f83a36-18fb-49b7-9914-87c8517d200e) } rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
2082025-05-08T22:00:08.649ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
2092025-05-08T22:00:08.649ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
2102025-05-08T22:00:08.658ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
2112025-05-08T22:00:08.658ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
2122025-05-08T22:00:08.659ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "3cebd119-9654-40e0-90a1-4e4a53a22158", "date": "Thu, 08 May 2025 22:00:08 GMT"} })
2132025-05-08T22:00:08.659ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
2142025-05-08T22:00:08.659ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
2152025-05-08T22:00:08.661ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
2162025-05-08T22:00:08.661ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
2172025-05-08T22:00:08.663ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
2182025-05-08T22:00:08.664ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
2192025-05-08T22:00:08.758ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1833 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
2202025-05-08T22:00:08.758ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1860 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 65f83a36-18fb-49b7-9914-87c8517d200e
2212025-05-08T22:00:08.758ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
2222025-05-08T22:00:08.758ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
2232025-05-08T22:00:08.758ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
2242025-05-08T22:00:08.758ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
2252025-05-08T22:00:08.758ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
2262025-05-08T22:00:08.758ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2007 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
2272025-05-08T22:00:08.758ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
2282025-05-08T22:00:08.758ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2043 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
2292025-05-08T22:00:08.758ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
2302025-05-08T22:00:08.758ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
2312025-05-08T22:00:08.759ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
2322025-05-08T22:00:08.761ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
2332025-05-08T22:00:08.761ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
2342025-05-08T22:00:08.777ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
2352025-05-08T22:00:08.777ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
2362025-05-08T22:00:08.790ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "41f08cb1-c5c1-4636-a50d-68c9189add02", "date": "Thu, 08 May 2025 22:00:08 GMT"} })
2372025-05-08T22:00:08.792ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 513.783796ms iteration = 4 status = {}
2382025-05-08T22:00:08.898ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "cbe24d18-beaa-47d0-9f58-396ed2f80b0a", "date": "Thu, 08 May 2025 22:00:08 GMT"} })
2392025-05-08T22:00:08.899ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch = Switch0
2402025-05-08T22:00:08.899ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
2412025-05-08T22:00:08.903ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "654310f5-563b-45f8-b5e1-8dfbe7c3a5e5", "content-length": "682", "date": "Thu, 08 May 2025 22:00:08 GMT"} })
2422025-05-08T22:00:08.903ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
2432025-05-08T22:00:08.903ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
2442025-05-08T22:00:08.907ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
2452025-05-08T22:00:08.907ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
2462025-05-08T22:00:08.910ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
2472025-05-08T22:00:08.911ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
2482025-05-08T22:00:08.914ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
2492025-05-08T22:00:08.914ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
2502025-05-08T22:00:08.921ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 642.129949ms iteration = 4 status = {}
2512025-05-08T22:00:11.934ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 317 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 86fff60d-6a74-4bec-97a9-de1e748ca3a4 response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
2522025-05-08T22:00:18.266ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 11 reason = Timeout
2532025-05-08T22:00:18.266ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2542025-05-08T22:00:18.266ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2552025-05-08T22:00:18.268ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2562025-05-08T22:00:18.271ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
2572025-05-08T22:00:18.271ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2582025-05-08T22:00:18.271ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2592025-05-08T22:00:18.273ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2602025-05-08T22:00:18.273ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2612025-05-08T22:00:18.275ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2622025-05-08T22:00:18.278ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
2632025-05-08T22:00:18.278ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2642025-05-08T22:00:18.278ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2652025-05-08T22:00:18.278ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 11 reason = Timeout
2662025-05-08T22:00:18.278ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
2672025-05-08T22:00:18.278ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
2682025-05-08T22:00:18.291ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
2692025-05-08T22:00:18.291ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 25.508162ms iteration = 11 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
2702025-05-08T22:00:18.315ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
2712025-05-08T22:00:18.315ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
2722025-05-08T22:00:18.358ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 79.512ms iteration = 11 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
2732025-05-08T22:00:21.934ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 256 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 08049429-0393-4f75-a514-001e3fa72b4a response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
2742025-05-08T22:00:28.266ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 12 reason = Timeout
2752025-05-08T22:00:28.266ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2762025-05-08T22:00:28.266ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2772025-05-08T22:00:28.268ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2782025-05-08T22:00:28.270ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
2792025-05-08T22:00:28.270ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2802025-05-08T22:00:28.271ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2812025-05-08T22:00:28.272ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2822025-05-08T22:00:28.273ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2832025-05-08T22:00:28.274ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2842025-05-08T22:00:28.277ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
2852025-05-08T22:00:28.277ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2862025-05-08T22:00:28.277ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2872025-05-08T22:00:28.279ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 12 reason = Timeout
2882025-05-08T22:00:28.279ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
2892025-05-08T22:00:28.279ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
2902025-05-08T22:00:28.281ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
2912025-05-08T22:00:28.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
2922025-05-08T22:00:28.288ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
2932025-05-08T22:00:28.288ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.44772ms iteration = 12 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
2942025-05-08T22:00:28.318ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 39.623164ms iteration = 12 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
2952025-05-08T22:00:31.933ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 263 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 2d189100-3bed-40cd-9787-7d40b55cf1fe response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
2962025-05-08T22:00:38.264ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 3 reason = Timeout
2972025-05-08T22:00:38.264ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 3 reason = Timeout
2982025-05-08T22:00:38.264ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
2992025-05-08T22:00:38.264ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
3002025-05-08T22:00:38.264ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
3012025-05-08T22:00:38.264ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 3 reason = Timeout
3022025-05-08T22:00:38.264ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 3 reason = Timeout
3032025-05-08T22:00:38.264ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
3042025-05-08T22:00:38.264ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
3052025-05-08T22:00:38.264ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 5 reason = Timeout
3062025-05-08T22:00:38.264ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 3 reason = Timeout
3072025-05-08T22:00:38.264ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-05-08 21:50:38.264804504 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
3082025-05-08T22:00:38.264ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
3092025-05-08T22:00:38.264ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
3102025-05-08T22:00:38.264ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 3 reason = Timeout
3112025-05-08T22:00:38.264ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 5 reason = Timeout
3122025-05-08T22:00:38.264ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 5 reason = Timeout
3132025-05-08T22:00:38.264ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 5 reason = Timeout
3142025-05-08T22:00:38.264ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
3152025-05-08T22:00:38.265ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
3162025-05-08T22:00:38.265ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
3172025-05-08T22:00:38.265ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
3182025-05-08T22:00:38.265ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
3192025-05-08T22:00:38.265ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("228ed1c1-3cc1-4055-aae9-5595ad856b86.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("8b245480-f91d-406b-bb08-850e1c3e3fc5.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c8f20f67-ea63-4dbc-b0f1-c30876daff82.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c8f20f67-ea63-4dbc-b0f1-c30876daff82.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 1014, tv_nsec: 8200457 } })
3202025-05-08T22:00:38.265ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("0ccabf41-81cd-45e7-bb26-335edaa2ee47.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("38a08dd6-6a75-4558-b4f8-df853f4d03d9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("38a08dd6-6a75-4558-b4f8-df853f4d03d9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::9))) }], valid_until: Instant { tv_sec: 1014, tv_nsec: 8236507 } })
3212025-05-08T22:00:38.265ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
3222025-05-08T22:00:38.265ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
3232025-05-08T22:00:38.265ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } dns_group = external
3242025-05-08T22:00:38.265ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 13 reason = Timeout
3252025-05-08T22:00:38.265ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 993.08\u{b5}s iteration = 3 status = {"addresses":["[fd00:1122:3344:101::8]:5353","[fd00:1122:3344:101::9]:5353"]}
3262025-05-08T22:00:38.265ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3272025-05-08T22:00:38.265ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 4 reason = Timeout
3282025-05-08T22:00:38.265ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 5 reason = Timeout
3292025-05-08T22:00:38.265ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:85 reason = blueprint disabled target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2
3302025-05-08T22:00:38.265ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 6.17\u{b5}s iteration = 4 status = {"enabled":false,"target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2"}
3312025-05-08T22:00:38.265ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
3322025-05-08T22:00:38.265ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.182452ms iteration = 3 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:1::1]:5353"]}
3332025-05-08T22:00:38.266ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3342025-05-08T22:00:38.266ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
3352025-05-08T22:00:38.266ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
3362025-05-08T22:00:38.266ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
3372025-05-08T22:00:38.267ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3382025-05-08T22:00:38.267ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
3392025-05-08T22:00:38.267ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 5 reason = Timeout
3402025-05-08T22:00:38.267ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
3412025-05-08T22:00:38.268ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
3422025-05-08T22:00:38.269ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
3432025-05-08T22:00:38.269ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
3442025-05-08T22:00:38.269ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
3452025-05-08T22:00:38.272ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-05-08 21:50:38.264804504 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
3462025-05-08T22:00:38.272ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 6.55664ms iteration = 3 status = {"expiration":"2025-05-08T21:50:38.264804504Z","pruned":[]}
3472025-05-08T22:00:38.276ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3482025-05-08T22:00:38.276ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1014, tv_nsec: 19141243 } })
3492025-05-08T22:00:38.276ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1014, tv_nsec: 19401176 } })
3502025-05-08T22:00:38.276ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163
3512025-05-08T22:00:38.276ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1163
3522025-05-08T22:00:38.278ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3532025-05-08T22:00:38.278ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 12.492203ms iteration = 5 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
3542025-05-08T22:00:38.279ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
3552025-05-08T22:00:38.279ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 13 reason = Timeout
3562025-05-08T22:00:38.279ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
3572025-05-08T22:00:38.279ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 5 reason = Timeout
3582025-05-08T22:00:38.279ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
3592025-05-08T22:00:38.279ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 5 reason = Timeout
3602025-05-08T22:00:38.279ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 6 reason = Timeout
3612025-05-08T22:00:38.279ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
3622025-05-08T22:00:38.279ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
3632025-05-08T22:00:38.279ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 5 reason = Timeout
3642025-05-08T22:00:38.279ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
3652025-05-08T22:00:38.279ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
3662025-05-08T22:00:38.279ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 3 reason = Timeout
3672025-05-08T22:00:38.279ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:53
3682025-05-08T22:00:38.279ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
3692025-05-08T22:00:38.279ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 5 reason = Timeout
3702025-05-08T22:00:38.279ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 5 reason = Timeout
3712025-05-08T22:00:38.279ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
3722025-05-08T22:00:38.279ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 5 reason = Timeout
3732025-05-08T22:00:38.279ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
3742025-05-08T22:00:38.279ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
3752025-05-08T22:00:38.279ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 3 reason = Timeout
3762025-05-08T22:00:38.279ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
3772025-05-08T22:00:38.279ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
3782025-05-08T22:00:38.279ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 5 reason = Timeout
3792025-05-08T22:00:38.279ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
3802025-05-08T22:00:38.279ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
3812025-05-08T22:00:38.279ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
3822025-05-08T22:00:38.279ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
3832025-05-08T22:00:38.279ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
3842025-05-08T22:00:38.279ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
3852025-05-08T22:00:38.279ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
3862025-05-08T22:00:38.279ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
3872025-05-08T22:00:38.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
3882025-05-08T22:00:38.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
3892025-05-08T22:00:38.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
3902025-05-08T22:00:38.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
3912025-05-08T22:00:38.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
3922025-05-08T22:00:38.281ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
3932025-05-08T22:00:38.281ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
3942025-05-08T22:00:38.281ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
3952025-05-08T22:00:38.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
3962025-05-08T22:00:38.281ZDEBGnexus (BackgroundTasks): activating background_task = webhook_deliverator iteration = 3 reason = Timeout
3972025-05-08T22:00:38.281ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 3 reason = Timeout
3982025-05-08T22:00:38.281ZDEBGnexus (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} }
3992025-05-08T22:00:38.281ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
4002025-05-08T22:00:38.281ZDEBGnexus (BackgroundTasks): activating background_task = webhook_dispatcher iteration = 3 reason = Timeout
4012025-05-08T22:00:38.281ZDEBGnexus (BackgroundTasks): roles background_task = webhook_dispatcher roles = RoleSet { roles: {} }
4022025-05-08T22:00:38.281ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 3 reason = Timeout
4032025-05-08T22:00:38.281ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
4042025-05-08T22:00:38.281ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 5 reason = Timeout
4052025-05-08T22:00:38.281ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
4062025-05-08T22:00:38.281ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 5 reason = Timeout
4072025-05-08T22:00:38.281ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
4082025-05-08T22:00:38.281ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 5 reason = Timeout
4092025-05-08T22:00:38.281ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
4102025-05-08T22:00:38.281ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 5 reason = Timeout
4112025-05-08T22:00:38.281ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
4122025-05-08T22:00:38.281ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 5 reason = Timeout
4132025-05-08T22:00:38.281ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
4142025-05-08T22:00:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
4152025-05-08T22:00:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database result = Ok(())
4162025-05-08T22:00:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
4172025-05-08T22:00:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
4182025-05-08T22:00:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database result = Ok(())
4192025-05-08T22:00:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
4202025-05-08T22:00:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
4212025-05-08T22:00:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
4222025-05-08T22:00:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
4232025-05-08T22:00:38.283ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4242025-05-08T22:00:38.284ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4252025-05-08T22:00:38.287ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
4262025-05-08T22:00:38.287ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
4272025-05-08T22:00:38.287ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
4282025-05-08T22:00:38.287ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
4292025-05-08T22:00:38.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
4302025-05-08T22:00:38.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
4312025-05-08T22:00:38.314ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 33.161875ms iteration = 3 status = {"errors":[],"found_port_ok":[]}
4322025-05-08T22:00:38.343ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4332025-05-08T22:00:38.346ZDEBGnexus (BackgroundTasks): activation complete background_task = webhook_deliverator elapsed = 64.50608ms iteration = 3 status = {"by_rx":{},"error":null}
4342025-05-08T22:00:38.347ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4352025-05-08T22:00:38.347ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
4362025-05-08T22:00:38.347ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4372025-05-08T22:00:38.347ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4382025-05-08T22:00:38.350ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4392025-05-08T22:00:38.351ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
4402025-05-08T22:00:38.352ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
4412025-05-08T22:00:38.352ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
4422025-05-08T22:00:38.352ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
4432025-05-08T22:00:38.352ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
4442025-05-08T22:00:38.352ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
4452025-05-08T22:00:38.352ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
4462025-05-08T22:00:38.353ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
4472025-05-08T22:00:38.355ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
4482025-05-08T22:00:38.355ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
4492025-05-08T22:00:38.355ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
4502025-05-08T22:00:38.356ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4512025-05-08T22:00:38.357ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 75.242814ms iteration = 5 status = {"errors":[],"garbage_collect_requested":[]}
4522025-05-08T22:00:38.360ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
4532025-05-08T22:00:38.361ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
4542025-05-08T22:00:38.361ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
4552025-05-08T22:00:38.364ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4562025-05-08T22:00:38.367ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
4572025-05-08T22:00:38.367ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
4582025-05-08T22:00:38.367ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
4592025-05-08T22:00:38.375ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4602025-05-08T22:00:38.376ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4612025-05-08T22:00:38.381ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4622025-05-08T22:00:38.383ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
4632025-05-08T22:00:38.383ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
4642025-05-08T22:00:38.384ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4652025-05-08T22:00:38.384ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
4662025-05-08T22:00:38.385ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
4672025-05-08T22:00:38.385ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
4682025-05-08T22:00:38.385ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
4692025-05-08T22:00:38.385ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
4702025-05-08T22:00:38.385ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
4712025-05-08T22:00:38.385ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
4722025-05-08T22:00:38.385ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
4732025-05-08T22:00:38.385ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
4742025-05-08T22:00:38.387ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4752025-05-08T22:00:38.387ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
4762025-05-08T22:00:38.388ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
4772025-05-08T22:00:38.389ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
4782025-05-08T22:00:38.389ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
4792025-05-08T22:00:38.389ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
4802025-05-08T22:00:38.391ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4812025-05-08T22:00:38.391ZDEBGnexus (BackgroundTasks): roles background_task = webhook_dispatcher roles = RoleSet { roles: {} }
4822025-05-08T22:00:38.391ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
4832025-05-08T22:00:38.391ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
4842025-05-08T22:00:38.391ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database result = Ok(())
4852025-05-08T22:00:38.391ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
4862025-05-08T22:00:38.394ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
4872025-05-08T22:00:38.394ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
4882025-05-08T22:00:38.394ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 112.675175ms iteration = 3 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
4892025-05-08T22:00:38.395ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
4902025-05-08T22:00:38.395ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
4912025-05-08T22:00:38.395ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
4922025-05-08T22:00:38.397ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 116.057821ms iteration = 5 status = {"errors":[],"requests_created_ok":[]}
4932025-05-08T22:00:38.399ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4942025-05-08T22:00:38.404ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
4952025-05-08T22:00:38.404ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4962025-05-08T22:00:38.404ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4972025-05-08T22:00:38.406ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
4982025-05-08T22:00:38.406ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
4992025-05-08T22:00:38.409ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5002025-05-08T22:00:38.409ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5012025-05-08T22:00:38.409ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
5022025-05-08T22:00:38.410ZDEBGnexus (BackgroundTasks): activation complete background_task = webhook_dispatcher elapsed = 128.728167ms iteration = 3 status = {"dispatched":[],"errors":[],"glob_version":"140.0.0","globs_reprocessed":{},"no_receivers":[]}
5032025-05-08T22:00:38.410ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1014, tv_nsec: 153756953 } })
5042025-05-08T22:00:38.411ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
5052025-05-08T22:00:38.411ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
5062025-05-08T22:00:38.411ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
5072025-05-08T22:00:38.411ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "1018f135-e4e5-413c-8a80-80231303f658", "content-length": "26", "date": "Thu, 08 May 2025 22:00:38 GMT"} })
5082025-05-08T22:00:38.412ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
5092025-05-08T22:00:38.414ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
5102025-05-08T22:00:38.415ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
5112025-05-08T22:00:38.420ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 140.933097ms iteration = 13 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
5122025-05-08T22:00:38.434ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
5132025-05-08T22:00:38.434ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
5142025-05-08T22:00:38.434ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
5152025-05-08T22:00:38.434ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
5162025-05-08T22:00:38.434ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
5172025-05-08T22:00:38.435ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
5182025-05-08T22:00:38.435ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
5192025-05-08T22:00:38.436ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5202025-05-08T22:00:38.437ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 155.22738ms iteration = 5 status = {"errors":[],"finish_invoked_ok":[]}
5212025-05-08T22:00:38.440ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
5222025-05-08T22:00:38.440ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
5232025-05-08T22:00:38.441ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
5242025-05-08T22:00:38.452ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5252025-05-08T22:00:38.452ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5262025-05-08T22:00:38.453ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
5272025-05-08T22:00:38.454ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
5282025-05-08T22:00:38.455ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5292025-05-08T22:00:38.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5302025-05-08T22:00:38.457ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5312025-05-08T22:00:38.457ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5322025-05-08T22:00:38.458ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:552 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
5332025-05-08T22:00:38.458ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 179.203616ms iteration = 5 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
5342025-05-08T22:00:38.460ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
5352025-05-08T22:00:38.460ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
5362025-05-08T22:00:38.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
5372025-05-08T22:00:38.462ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 2f2f2c32-8d58-4801-a669-9618eac0c3f7, lookup_type: ById(2f2f2c32-8d58-4801-a669-9618eac0c3f7) } result = Ok(())
5382025-05-08T22:00:38.462ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5392025-05-08T22:00:38.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5402025-05-08T22:00:38.463ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 184.049608ms iteration = 5 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
5412025-05-08T22:00:38.463ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 181.881195ms iteration = 5 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
5422025-05-08T22:00:38.465ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 185.546024ms iteration = 5 status = 0
5432025-05-08T22:00:38.471ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5442025-05-08T22:00:38.475ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
5452025-05-08T22:00:38.475ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
5462025-05-08T22:00:38.475ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
5472025-05-08T22:00:38.479ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
5482025-05-08T22:00:38.479ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
5492025-05-08T22:00:38.479ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
5502025-05-08T22:00:38.479ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
5512025-05-08T22:00:38.480ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
5522025-05-08T22:00:38.480ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = d8128696-db5a-4b88-914a-69b28494e50e dns_zone_name = oxide.test found_so_far = 0 version = 2
5532025-05-08T22:00:38.480ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
5542025-05-08T22:00:38.480ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
5552025-05-08T22:00:38.480ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5562025-05-08T22:00:38.482ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5572025-05-08T22:00:38.482ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
5582025-05-08T22:00:38.482ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
5592025-05-08T22:00:38.483ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
5602025-05-08T22:00:38.484ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
5612025-05-08T22:00:38.484ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:137 found = 0 not_ready_to_be_deleted = 0
5622025-05-08T22:00:38.484ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 204.973092ms iteration = 3 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
5632025-05-08T22:00:38.485ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
5642025-05-08T22:00:38.486ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
5652025-05-08T22:00:38.486ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
5662025-05-08T22:00:38.495ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = d8128696-db5a-4b88-914a-69b28494e50e dns_zone_name = oxide.test found_so_far = 1 version = 2
5672025-05-08T22:00:38.495ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
5682025-05-08T22:00:38.495ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-05-08 21:59:21.759368 UTC dns_group = external generation = 2
5692025-05-08T22:00:38.495ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 230.631676ms iteration = 3 status = {"generation":2}
5702025-05-08T22:00:38.500ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
5712025-05-08T22:00:38.500ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 7586992a-6fc5-4073-8962-728778f16aa5 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
5722025-05-08T22:00:38.500ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
5732025-05-08T22:00:38.500ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
5742025-05-08T22:00:38.505ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5752025-05-08T22:00:38.510ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
5762025-05-08T22:00:38.510ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
5772025-05-08T22:00:38.510ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
5782025-05-08T22:00:38.514ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 7586992a-6fc5-4073-8962-728778f16aa5 dns_zone_name = control-plane.oxide.internal found_so_far = 55 version = 1
5792025-05-08T22:00:38.514ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
5802025-05-08T22:00:38.514ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-05-08 21:59:21.182334 UTC dns_group = internal generation = 1
5812025-05-08T22:00:38.514ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 249.614369ms iteration = 3 status = {"generation":1}
5822025-05-08T22:00:38.514ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-05-08T21:58:35.560911Z, time_modified: 2025-05-08T21:58:35.560911Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, db_silo: Silo { identity: SiloIdentity { id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-05-08T21:59:21.709488Z, time_modified: 2025-05-08T21:59:21.709488Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, db_silo: Silo { identity: SiloIdentity { id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-05-08T21:59:21.709488Z, time_modified: 2025-05-08T21:59:21.709488Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47" }] }) }
5832025-05-08T22:00:38.515ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 250.355087ms iteration = 3 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"7137ee58-6aa7-4199-a248-feadf3b8bafb","tls_certs":["37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47"]}},"default_endpoint":{"silo_id":"7137ee58-6aa7-4199-a248-feadf3b8bafb","tls_certs":["37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
5842025-05-08T22:00:38.515ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
5852025-05-08T22:00:38.516ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
5862025-05-08T22:00:38.527ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
5872025-05-08T22:00:38.527ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
5882025-05-08T22:00:38.529ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
5892025-05-08T22:00:38.529ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 263.446738ms iteration = 13 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
5902025-05-08T22:00:38.532ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
5912025-05-08T22:00:38.532ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:362
5922025-05-08T22:00:38.532ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 264.747922ms iteration = 5 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
5932025-05-08T22:00:38.533ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
5942025-05-08T22:00:38.535ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
5952025-05-08T22:00:38.535ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
5962025-05-08T22:00:38.535ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 256.350081ms iteration = 5 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
5972025-05-08T22:00:38.565ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
5982025-05-08T22:00:38.565ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
5992025-05-08T22:00:38.565ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e zone_address = fd00:1122:3344:101::2
6002025-05-08T22:00:38.565ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
6012025-05-08T22:00:38.565ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "23eaa81b-de58-4ba2-a5d6-220cfbed4877", "content-length": "26", "date": "Thu, 08 May 2025 22:00:38 GMT"} })
6022025-05-08T22:00:38.565ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
6032025-05-08T22:00:38.565ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "381b096a-b4d6-41b0-9aeb-49cc4e434ec0", "content-length": "26", "date": "Thu, 08 May 2025 22:00:38 GMT"} })
6042025-05-08T22:00:38.565ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1174 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
6052025-05-08T22:00:38.572ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
6062025-05-08T22:00:38.573ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
6072025-05-08T22:00:38.578ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
6082025-05-08T22:00:38.578ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
6092025-05-08T22:00:38.581ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 316.604356ms iteration = 5 status = null
6102025-05-08T22:00:38.591ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 309.225317ms iteration = 5 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
6112025-05-08T22:00:38.604ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
6122025-05-08T22:00:38.604ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "49ad2292-ed66-455b-9c4e-e0d6c923b6ff", "date": "Thu, 08 May 2025 22:00:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
6132025-05-08T22:00:38.604ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
6142025-05-08T22:00:38.606ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "cc7d1629-2df8-49a4-9589-0e26762c59bd", "date": "Thu, 08 May 2025 22:00:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
6152025-05-08T22:00:38.606ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
6162025-05-08T22:00:38.608ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
6172025-05-08T22:00:38.608ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 343.281021ms iteration = 5 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
6182025-05-08T22:00:38.674ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 394.832333ms iteration = 6 status = {"physical_disks_added":0}
6192025-05-08T22:00:38.695ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
6202025-05-08T22:00:38.709ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 444.917829ms iteration = 5 status = {}
6212025-05-08T22:00:38.737ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
6222025-05-08T22:00:38.737ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
6232025-05-08T22:00:38.739ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6242025-05-08T22:00:38.742ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
6252025-05-08T22:00:38.742ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
6262025-05-08T22:00:38.743ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
6272025-05-08T22:00:38.744ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6282025-05-08T22:00:38.747ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
6292025-05-08T22:00:38.747ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
6302025-05-08T22:00:38.747ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
6312025-05-08T22:00:38.750ZINFOnexus (BackgroundTasks): instance reincarnation completed successfully background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:113 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0
6322025-05-08T22:00:38.750ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 471.058909ms iteration = 3 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
6332025-05-08T22:00:38.757ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
6342025-05-08T22:00:38.757ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
6352025-05-08T22:00:38.758ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "9cad191d-7faa-4291-ad9b-e465c0842a2e", "content-length": "372", "date": "Thu, 08 May 2025 22:00:38 GMT"} })
6362025-05-08T22:00:38.758ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
6372025-05-08T22:00:38.758ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
6382025-05-08T22:00:38.830ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
6392025-05-08T22:00:38.830ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
6402025-05-08T22:00:38.830ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "8bb55f5a-ffbd-4db6-8ba1-fbe2785680f6", "date": "Thu, 08 May 2025 22:00:38 GMT"} })
6412025-05-08T22:00:38.830ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
6422025-05-08T22:00:38.831ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
6432025-05-08T22:00:38.868ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
6442025-05-08T22:00:38.869ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "fd5061f8-4f27-4f09-9a52-75d5776d7d6a", "date": "Thu, 08 May 2025 22:00:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
6452025-05-08T22:00:38.869ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
6462025-05-08T22:00:38.870ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "692762a4-55ce-41c5-a38b-2c065d49bb7b", "date": "Thu, 08 May 2025 22:00:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
6472025-05-08T22:00:38.870ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
6482025-05-08T22:00:38.872ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "052ce8cc-17ec-47c8-9145-ca0e15c04ef5", "date": "Thu, 08 May 2025 22:00:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
6492025-05-08T22:00:38.872ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
6502025-05-08T22:00:38.874ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
6512025-05-08T22:00:38.874ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 609.58598ms iteration = 5 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
6522025-05-08T22:00:38.902ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
6532025-05-08T22:00:38.902ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "7eedfce8-18ad-4084-8542-bc8ccd35bed7", "content-length": "691", "date": "Thu, 08 May 2025 22:00:38 GMT"} })
6542025-05-08T22:00:38.902ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
6552025-05-08T22:00:38.902ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
6562025-05-08T22:00:38.902ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
6572025-05-08T22:00:38.905ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 625.688522ms iteration = 5 status = {}
6582025-05-08T22:00:38.931ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
6592025-05-08T22:00:38.931ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
6602025-05-08T22:00:38.933ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: cd0b930c-534b-46ad-ac3b-2eaa94b0bf2c, rack_id: 64f3d6f4-89b8-48aa-a622-b684e261667e, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(65f83a36-18fb-49b7-9914-87c8517d200e) } rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
6612025-05-08T22:00:38.933ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
6622025-05-08T22:00:38.933ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
6632025-05-08T22:00:38.936ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
6642025-05-08T22:00:38.937ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
6652025-05-08T22:00:39.046ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
6662025-05-08T22:00:39.046ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
6672025-05-08T22:00:39.050ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1833 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
6682025-05-08T22:00:39.050ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1860 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 65f83a36-18fb-49b7-9914-87c8517d200e
6692025-05-08T22:00:39.050ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
6702025-05-08T22:00:39.050ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
6712025-05-08T22:00:39.050ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
6722025-05-08T22:00:39.050ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
6732025-05-08T22:00:39.050ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
6742025-05-08T22:00:39.050ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
6752025-05-08T22:00:39.050ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
6762025-05-08T22:00:39.050ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2007 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
6772025-05-08T22:00:39.050ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
6782025-05-08T22:00:39.050ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2043 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
6792025-05-08T22:00:39.050ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "acec2026-51ed-42fd-bfca-ff21fb28777a", "date": "Thu, 08 May 2025 22:00:38 GMT"} })
6802025-05-08T22:00:39.052ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 773.260811ms iteration = 5 status = {}
6812025-05-08T22:00:39.052ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
6822025-05-08T22:00:39.052ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
6832025-05-08T22:00:39.052ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
6842025-05-08T22:00:39.055ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
6852025-05-08T22:00:39.055ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
6862025-05-08T22:00:39.230ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "a7d617dd-9bf5-4602-9b69-024d9c56d2bb", "date": "Thu, 08 May 2025 22:00:38 GMT"} })
6872025-05-08T22:00:39.230ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch = Switch0
6882025-05-08T22:00:39.230ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
6892025-05-08T22:00:39.231ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8867925-62f8-4bd2-a377-7eadfeb9837b", "content-length": "682", "date": "Thu, 08 May 2025 22:00:39 GMT"} })
6902025-05-08T22:00:39.231ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
6912025-05-08T22:00:39.231ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
6922025-05-08T22:00:39.234ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
6932025-05-08T22:00:39.235ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
6942025-05-08T22:00:39.238ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
6952025-05-08T22:00:39.238ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
6962025-05-08T22:00:39.240ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
6972025-05-08T22:00:39.240ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
6982025-05-08T22:00:39.247ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 968.344718ms iteration = 5 status = {}
6992025-05-08T22:00:41.934ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 253 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 2a7f1ec1-8a26-4c51-a550-2099b21c6dad response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
7002025-05-08T22:00:48.266ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 14 reason = Timeout
7012025-05-08T22:00:48.266ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7022025-05-08T22:00:48.266ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7032025-05-08T22:00:48.268ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7042025-05-08T22:00:48.271ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
7052025-05-08T22:00:48.271ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7062025-05-08T22:00:48.271ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7072025-05-08T22:00:48.272ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7082025-05-08T22:00:48.273ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7092025-05-08T22:00:48.274ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7102025-05-08T22:00:48.276ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
7112025-05-08T22:00:48.276ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7122025-05-08T22:00:48.277ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7132025-05-08T22:00:48.278ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 14 reason = Timeout
7142025-05-08T22:00:48.278ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
7152025-05-08T22:00:48.279ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
7162025-05-08T22:00:48.280ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
7172025-05-08T22:00:48.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
7182025-05-08T22:00:48.282ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.71276ms iteration = 14 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
7192025-05-08T22:00:48.484ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
7202025-05-08T22:00:48.484ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 217.729839ms iteration = 14 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
7212025-05-08T22:00:51.934ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 280 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 758835f2-6762-4348-909d-7aba25dd7cde response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
7222025-05-08T22:00:58.266ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 15 reason = Timeout
7232025-05-08T22:00:58.266ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7242025-05-08T22:00:58.266ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7252025-05-08T22:00:58.268ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7262025-05-08T22:00:58.271ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
7272025-05-08T22:00:58.271ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7282025-05-08T22:00:58.271ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7292025-05-08T22:00:58.273ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7302025-05-08T22:00:58.273ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7312025-05-08T22:00:58.275ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7322025-05-08T22:00:58.278ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
7332025-05-08T22:00:58.278ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7342025-05-08T22:00:58.278ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7352025-05-08T22:00:58.279ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 15 reason = Timeout
7362025-05-08T22:00:58.279ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
7372025-05-08T22:00:58.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
7382025-05-08T22:00:58.281ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
7392025-05-08T22:00:58.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
7402025-05-08T22:00:58.314ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 35.076036ms iteration = 15 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
7412025-05-08T22:00:58.491ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
7422025-05-08T22:00:58.491ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 225.401661ms iteration = 15 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
7432025-05-08T22:01:01.934ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 332 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 264059f7-c657-40ce-83f9-62b2ba9b6e58 response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
7442025-05-08T22:01:08.264ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 6 reason = Timeout
7452025-05-08T22:01:08.264ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
7462025-05-08T22:01:08.264ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 6 reason = Timeout
7472025-05-08T22:01:08.264ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
7482025-05-08T22:01:08.265ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
7492025-05-08T22:01:08.265ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
7502025-05-08T22:01:08.266ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 16 reason = Timeout
7512025-05-08T22:01:08.266ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 6 reason = Timeout
7522025-05-08T22:01:08.266ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7532025-05-08T22:01:08.266ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7542025-05-08T22:01:08.267ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1044, tv_nsec: 10558662 } })
7552025-05-08T22:01:08.268ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 6 reason = Timeout
7562025-05-08T22:01:08.268ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
7572025-05-08T22:01:08.268ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
7582025-05-08T22:01:08.268ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163
7592025-05-08T22:01:08.268ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1044, tv_nsec: 11127638 } })
7602025-05-08T22:01:08.268ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1163
7612025-05-08T22:01:08.268ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7622025-05-08T22:01:08.269ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7632025-05-08T22:01:08.270ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.321466ms iteration = 6 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
7642025-05-08T22:01:08.271ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
7652025-05-08T22:01:08.271ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7662025-05-08T22:01:08.271ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7672025-05-08T22:01:08.272ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
7682025-05-08T22:01:08.272ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
7692025-05-08T22:01:08.272ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
7702025-05-08T22:01:08.273ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7712025-05-08T22:01:08.274ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7722025-05-08T22:01:08.275ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7732025-05-08T22:01:08.278ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
7742025-05-08T22:01:08.278ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7752025-05-08T22:01:08.278ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7762025-05-08T22:01:08.279ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 16 reason = Timeout
7772025-05-08T22:01:08.280ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
7782025-05-08T22:01:08.280ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 6 reason = Timeout
7792025-05-08T22:01:08.280ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
7802025-05-08T22:01:08.280ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 6 reason = Timeout
7812025-05-08T22:01:08.280ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
7822025-05-08T22:01:08.280ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 6 reason = Timeout
7832025-05-08T22:01:08.280ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
7842025-05-08T22:01:08.280ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
7852025-05-08T22:01:08.280ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 6 reason = Timeout
7862025-05-08T22:01:08.280ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
7872025-05-08T22:01:08.280ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 6 reason = Timeout
7882025-05-08T22:01:08.280ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
7892025-05-08T22:01:08.280ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
7902025-05-08T22:01:08.280ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 6 reason = Timeout
7912025-05-08T22:01:08.280ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 6 reason = Timeout
7922025-05-08T22:01:08.280ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
7932025-05-08T22:01:08.280ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
7942025-05-08T22:01:08.280ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 7 reason = Timeout
7952025-05-08T22:01:08.280ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
7962025-05-08T22:01:08.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
7972025-05-08T22:01:08.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
7982025-05-08T22:01:08.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
7992025-05-08T22:01:08.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
8002025-05-08T22:01:08.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
8012025-05-08T22:01:08.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
8022025-05-08T22:01:08.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
8032025-05-08T22:01:08.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
8042025-05-08T22:01:08.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
8052025-05-08T22:01:08.281ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 6 reason = Timeout
8062025-05-08T22:01:08.281ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 6 reason = Timeout
8072025-05-08T22:01:08.281ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
8082025-05-08T22:01:08.281ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
8092025-05-08T22:01:08.281ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 6 reason = Timeout
8102025-05-08T22:01:08.281ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
8112025-05-08T22:01:08.281ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 6 reason = Timeout
8122025-05-08T22:01:08.281ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
8132025-05-08T22:01:08.281ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 6 reason = Timeout
8142025-05-08T22:01:08.281ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
8152025-05-08T22:01:08.281ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
8162025-05-08T22:01:08.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
8172025-05-08T22:01:08.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
8182025-05-08T22:01:08.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
8192025-05-08T22:01:08.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
8202025-05-08T22:01:08.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
8212025-05-08T22:01:08.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
8222025-05-08T22:01:08.282ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8232025-05-08T22:01:08.283ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8242025-05-08T22:01:08.284ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8252025-05-08T22:01:08.284ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8262025-05-08T22:01:08.287ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
8272025-05-08T22:01:08.287ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
8282025-05-08T22:01:08.287ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
8292025-05-08T22:01:08.287ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
8302025-05-08T22:01:08.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
8312025-05-08T22:01:08.287ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
8322025-05-08T22:01:08.287ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
8332025-05-08T22:01:08.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
8342025-05-08T22:01:08.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
8352025-05-08T22:01:08.291ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
8362025-05-08T22:01:08.291ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
8372025-05-08T22:01:08.291ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
8382025-05-08T22:01:08.295ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8392025-05-08T22:01:08.299ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
8402025-05-08T22:01:08.299ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
8412025-05-08T22:01:08.299ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
8422025-05-08T22:01:08.300ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
8432025-05-08T22:01:08.300ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
8442025-05-08T22:01:08.300ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
8452025-05-08T22:01:08.301ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
8462025-05-08T22:01:08.302ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 20.821392ms iteration = 6 status = {"errors":[],"finish_invoked_ok":[]}
8472025-05-08T22:01:08.302ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
8482025-05-08T22:01:08.303ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
8492025-05-08T22:01:08.308ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
8502025-05-08T22:01:08.308ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
8512025-05-08T22:01:08.308ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
8522025-05-08T22:01:08.308ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
8532025-05-08T22:01:08.336ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
8542025-05-08T22:01:08.337ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
8552025-05-08T22:01:08.345ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
8562025-05-08T22:01:08.345ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
8572025-05-08T22:01:08.345ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "41ddc6c8-587f-4027-a3de-c80bc4cd97c8", "content-length": "26", "date": "Thu, 08 May 2025 22:01:08 GMT"} })
8582025-05-08T22:01:08.345ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
8592025-05-08T22:01:08.347ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
8602025-05-08T22:01:08.358ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 78.605351ms iteration = 16 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
8612025-05-08T22:01:08.360ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
8622025-05-08T22:01:08.360ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
8632025-05-08T22:01:08.360ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
8642025-05-08T22:01:08.360ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
8652025-05-08T22:01:08.364ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 82.661564ms iteration = 6 status = {"errors":[],"garbage_collect_requested":[]}
8662025-05-08T22:01:08.364ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
8672025-05-08T22:01:08.364ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
8682025-05-08T22:01:08.364ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "629119df-e3f2-4176-95bd-bae22d1d9517", "content-length": "26", "date": "Thu, 08 May 2025 22:01:08 GMT"} })
8692025-05-08T22:01:08.364ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
8702025-05-08T22:01:08.365ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 83.579754ms iteration = 6 status = {"errors":[],"requests_created_ok":[]}
8712025-05-08T22:01:08.366ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
8722025-05-08T22:01:08.369ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
8732025-05-08T22:01:08.369ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8742025-05-08T22:01:08.369ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
8752025-05-08T22:01:08.369ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8762025-05-08T22:01:08.371ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
8772025-05-08T22:01:08.371ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
8782025-05-08T22:01:08.372ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
8792025-05-08T22:01:08.372ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
8802025-05-08T22:01:08.372ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
8812025-05-08T22:01:08.373ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
8822025-05-08T22:01:08.373ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
8832025-05-08T22:01:08.373ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
8842025-05-08T22:01:08.377ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
8852025-05-08T22:01:08.377ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 97.511663ms iteration = 6 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
8862025-05-08T22:01:08.377ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
8872025-05-08T22:01:08.383ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:552 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
8882025-05-08T22:01:08.383ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 103.136383ms iteration = 6 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
8892025-05-08T22:01:08.395ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8902025-05-08T22:01:08.398ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
8912025-05-08T22:01:08.398ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
8922025-05-08T22:01:08.398ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
8932025-05-08T22:01:08.415ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
8942025-05-08T22:01:08.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
8952025-05-08T22:01:08.418ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 153.372141ms iteration = 6 status = null
8962025-05-08T22:01:08.420ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1044, tv_nsec: 163446427 } })
8972025-05-08T22:01:08.421ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
8982025-05-08T22:01:08.426ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 145.530127ms iteration = 6 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
8992025-05-08T22:01:08.428ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
9002025-05-08T22:01:08.428ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
9012025-05-08T22:01:08.428ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 148.511738ms iteration = 6 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
9022025-05-08T22:01:08.433ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 168.892637ms iteration = 6 status = {}
9032025-05-08T22:01:08.437ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 155.887977ms iteration = 6 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
9042025-05-08T22:01:08.447ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:362
9052025-05-08T22:01:08.447ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 179.321898ms iteration = 6 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
9062025-05-08T22:01:08.455ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9072025-05-08T22:01:08.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9082025-05-08T22:01:08.457ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e zone_address = fd00:1122:3344:101::2
9092025-05-08T22:01:08.457ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
9102025-05-08T22:01:08.457ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9112025-05-08T22:01:08.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9122025-05-08T22:01:08.458ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "54a0e683-be2a-48c3-894a-2deacddc5e56", "content-length": "26", "date": "Thu, 08 May 2025 22:01:08 GMT"} })
9132025-05-08T22:01:08.458ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1174 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
9142025-05-08T22:01:08.459ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9152025-05-08T22:01:08.459ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
9162025-05-08T22:01:08.462ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 2f2f2c32-8d58-4801-a669-9618eac0c3f7, lookup_type: ById(2f2f2c32-8d58-4801-a669-9618eac0c3f7) } result = Ok(())
9172025-05-08T22:01:08.462ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9182025-05-08T22:01:08.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9192025-05-08T22:01:08.464ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 184.387913ms iteration = 6 status = 0
9202025-05-08T22:01:08.466ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
9212025-05-08T22:01:08.466ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
9222025-05-08T22:01:08.500ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
9232025-05-08T22:01:08.500ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 234.104964ms iteration = 16 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
9242025-05-08T22:01:08.519ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 239.461561ms iteration = 7 status = {"physical_disks_added":0}
9252025-05-08T22:01:08.551ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
9262025-05-08T22:01:08.551ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
9272025-05-08T22:01:08.552ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "15d75c89-7f05-4510-8ae4-8b6cb65d072e", "content-length": "372", "date": "Thu, 08 May 2025 22:01:08 GMT"} })
9282025-05-08T22:01:08.552ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
9292025-05-08T22:01:08.552ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
9302025-05-08T22:01:08.585ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
9312025-05-08T22:01:08.585ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
9322025-05-08T22:01:08.585ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "f34f7ba9-26b2-42e3-a8ec-7b56e69ab1ab", "date": "Thu, 08 May 2025 22:01:08 GMT"} })
9332025-05-08T22:01:08.585ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
9342025-05-08T22:01:08.585ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
9352025-05-08T22:01:08.588ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
9362025-05-08T22:01:08.588ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
9372025-05-08T22:01:08.591ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
9382025-05-08T22:01:08.592ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
9392025-05-08T22:01:08.619ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
9402025-05-08T22:01:08.620ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
9412025-05-08T22:01:08.622ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: cd0b930c-534b-46ad-ac3b-2eaa94b0bf2c, rack_id: 64f3d6f4-89b8-48aa-a622-b684e261667e, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(65f83a36-18fb-49b7-9914-87c8517d200e) } rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
9422025-05-08T22:01:08.622ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
9432025-05-08T22:01:08.622ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
9442025-05-08T22:01:08.637ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
9452025-05-08T22:01:08.638ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
9462025-05-08T22:01:08.640ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1833 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
9472025-05-08T22:01:08.640ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1860 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 65f83a36-18fb-49b7-9914-87c8517d200e
9482025-05-08T22:01:08.640ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9492025-05-08T22:01:08.641ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
9502025-05-08T22:01:08.641ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
9512025-05-08T22:01:08.641ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
9522025-05-08T22:01:08.641ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
9532025-05-08T22:01:08.641ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
9542025-05-08T22:01:08.641ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2007 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
9552025-05-08T22:01:08.641ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
9562025-05-08T22:01:08.641ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2043 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
9572025-05-08T22:01:08.641ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
9582025-05-08T22:01:08.641ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
9592025-05-08T22:01:08.641ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
9602025-05-08T22:01:08.643ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
9612025-05-08T22:01:08.643ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
9622025-05-08T22:01:08.643ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
9632025-05-08T22:01:08.645ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9642025-05-08T22:01:08.646ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
9652025-05-08T22:01:08.646ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
9662025-05-08T22:01:08.648ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
9672025-05-08T22:01:08.649ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
9682025-05-08T22:01:08.649ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
9692025-05-08T22:01:08.689ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
9702025-05-08T22:01:08.689ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
9712025-05-08T22:01:08.754ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "5c3ef089-86dc-4645-97b8-7592eb0ba395", "date": "Thu, 08 May 2025 22:01:08 GMT"} })
9722025-05-08T22:01:08.754ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch = Switch0
9732025-05-08T22:01:08.754ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
9742025-05-08T22:01:08.755ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e74d8c2-2129-4ccd-a615-12e1113dd9e5", "content-length": "682", "date": "Thu, 08 May 2025 22:01:08 GMT"} })
9752025-05-08T22:01:08.755ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
9762025-05-08T22:01:08.755ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
9772025-05-08T22:01:08.755ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "885be006-4c16-4123-bf41-1f262532d4f1", "date": "Thu, 08 May 2025 22:01:08 GMT"} })
9782025-05-08T22:01:08.757ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 477.478937ms iteration = 6 status = {}
9792025-05-08T22:01:08.759ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
9802025-05-08T22:01:08.760ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
9812025-05-08T22:01:08.763ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
9822025-05-08T22:01:08.763ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
9832025-05-08T22:01:08.765ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
9842025-05-08T22:01:08.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
9852025-05-08T22:01:08.773ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 493.021964ms iteration = 6 status = {}
9862025-05-08T22:01:08.852ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
9872025-05-08T22:01:08.853ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "2067f4d4-7299-45bf-82a0-8b4fb9dfa15a", "content-length": "691", "date": "Thu, 08 May 2025 22:01:08 GMT"} })
9882025-05-08T22:01:08.853ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
9892025-05-08T22:01:08.853ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
9902025-05-08T22:01:08.853ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
9912025-05-08T22:01:08.854ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 574.0761ms iteration = 6 status = {}
9922025-05-08T22:01:11.934ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 460 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = c4a0f16f-6c12-4d54-90e7-ad75477f9317 response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
9932025-05-08T22:01:18.266ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 17 reason = Timeout
9942025-05-08T22:01:18.266ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9952025-05-08T22:01:18.266ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9962025-05-08T22:01:18.268ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9972025-05-08T22:01:18.271ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
9982025-05-08T22:01:18.271ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9992025-05-08T22:01:18.271ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10002025-05-08T22:01:18.273ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10012025-05-08T22:01:18.273ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10022025-05-08T22:01:18.275ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10032025-05-08T22:01:18.278ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
10042025-05-08T22:01:18.278ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10052025-05-08T22:01:18.278ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10062025-05-08T22:01:18.279ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 17 reason = Timeout
10072025-05-08T22:01:18.279ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
10082025-05-08T22:01:18.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
10092025-05-08T22:01:18.282ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
10102025-05-08T22:01:18.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
10112025-05-08T22:01:18.288ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
10122025-05-08T22:01:18.288ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.118716ms iteration = 17 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
10132025-05-08T22:01:18.314ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 34.373217ms iteration = 17 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
10142025-05-08T22:01:21.934ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 253 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = aea83d0d-6a8f-4c01-834c-f3b959596042 response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
10152025-05-08T22:01:28.266ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 18 reason = Timeout
10162025-05-08T22:01:28.266ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10172025-05-08T22:01:28.266ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10182025-05-08T22:01:28.268ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10192025-05-08T22:01:28.271ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
10202025-05-08T22:01:28.271ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10212025-05-08T22:01:28.271ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10222025-05-08T22:01:28.273ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10232025-05-08T22:01:28.274ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10242025-05-08T22:01:28.275ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10252025-05-08T22:01:28.277ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
10262025-05-08T22:01:28.277ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10272025-05-08T22:01:28.278ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10282025-05-08T22:01:28.280ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 18 reason = Timeout
10292025-05-08T22:01:28.280ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
10302025-05-08T22:01:28.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
10312025-05-08T22:01:28.282ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
10322025-05-08T22:01:28.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
10332025-05-08T22:01:28.283ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.590019ms iteration = 18 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
10342025-05-08T22:01:28.290ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
10352025-05-08T22:01:28.290ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 23.788154ms iteration = 18 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
10362025-05-08T22:01:31.935ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 300 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 06f50309-5caa-47c7-9571-0d82b5b8f44d response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
10372025-05-08T22:01:38.264ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 7 reason = Timeout
10382025-05-08T22:01:38.264ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
10392025-05-08T22:01:38.264ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 6 reason = Timeout
10402025-05-08T22:01:38.264ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 4 reason = Timeout
10412025-05-08T22:01:38.264ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
10422025-05-08T22:01:38.264ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 4 reason = Timeout
10432025-05-08T22:01:38.264ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 6 reason = Timeout
10442025-05-08T22:01:38.264ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 4 reason = Timeout
10452025-05-08T22:01:38.264ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
10462025-05-08T22:01:38.264ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 4 reason = Timeout
10472025-05-08T22:01:38.264ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
10482025-05-08T22:01:38.264ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 4 reason = Timeout
10492025-05-08T22:01:38.264ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
10502025-05-08T22:01:38.264ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
10512025-05-08T22:01:38.264ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 7 reason = Timeout
10522025-05-08T22:01:38.264ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 4 reason = Timeout
10532025-05-08T22:01:38.264ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
10542025-05-08T22:01:38.264ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-05-08 21:51:38.264589437 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
10552025-05-08T22:01:38.264ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
10562025-05-08T22:01:38.264ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
10572025-05-08T22:01:38.264ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
10582025-05-08T22:01:38.265ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
10592025-05-08T22:01:38.265ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
10602025-05-08T22:01:38.265ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("0ccabf41-81cd-45e7-bb26-335edaa2ee47.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("38a08dd6-6a75-4558-b4f8-df853f4d03d9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("38a08dd6-6a75-4558-b4f8-df853f4d03d9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::9))) }], valid_until: Instant { tv_sec: 1074, tv_nsec: 7509099 } })
10612025-05-08T22:01:38.265ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
10622025-05-08T22:01:38.265ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
10632025-05-08T22:01:38.265ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("228ed1c1-3cc1-4055-aae9-5595ad856b86.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("8b245480-f91d-406b-bb08-850e1c3e3fc5.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c8f20f67-ea63-4dbc-b0f1-c30876daff82.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c8f20f67-ea63-4dbc-b0f1-c30876daff82.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 1074, tv_nsec: 7554250 } })
10642025-05-08T22:01:38.265ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } dns_group = external
10652025-05-08T22:01:38.265ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.106182ms iteration = 4 status = {"addresses":["[fd00:1122:3344:101::8]:5353","[fd00:1122:3344:101::9]:5353"]}
10662025-05-08T22:01:38.265ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
10672025-05-08T22:01:38.265ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.151512ms iteration = 4 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:1::1]:5353"]}
10682025-05-08T22:01:38.266ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 19 reason = Timeout
10692025-05-08T22:01:38.266ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 5 reason = Timeout
10702025-05-08T22:01:38.266ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:85 reason = blueprint disabled target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2
10712025-05-08T22:01:38.266ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10722025-05-08T22:01:38.266ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 9.92\u{b5}s iteration = 5 status = {"enabled":false,"target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2"}
10732025-05-08T22:01:38.266ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 7 reason = Timeout
10742025-05-08T22:01:38.266ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10752025-05-08T22:01:38.267ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
10762025-05-08T22:01:38.267ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10772025-05-08T22:01:38.267ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
10782025-05-08T22:01:38.267ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
10792025-05-08T22:01:38.267ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
10802025-05-08T22:01:38.268ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1074, tv_nsec: 10528771 } })
10812025-05-08T22:01:38.268ZDEBGnexus (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 3 reason = Timeout
10822025-05-08T22:01:38.268ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 7 reason = Timeout
10832025-05-08T22:01:38.268ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
10842025-05-08T22:01:38.268ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
10852025-05-08T22:01:38.268ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1163
10862025-05-08T22:01:38.268ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
10872025-05-08T22:01:38.268ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
10882025-05-08T22:01:38.268ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
10892025-05-08T22:01:38.269ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
10902025-05-08T22:01:38.269ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
10912025-05-08T22:01:38.269ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
10922025-05-08T22:01:38.269ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
10932025-05-08T22:01:38.269ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
10942025-05-08T22:01:38.269ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
10952025-05-08T22:01:38.269ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
10962025-05-08T22:01:38.270ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
10972025-05-08T22:01:38.270ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
10982025-05-08T22:01:38.270ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
10992025-05-08T22:01:38.270ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-05-08 21:51:38.264589437 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
11002025-05-08T22:01:38.270ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 5.802152ms iteration = 4 status = {"expiration":"2025-05-08T21:51:38.264589437Z","pruned":[]}
11012025-05-08T22:01:38.273ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11022025-05-08T22:01:38.274ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11032025-05-08T22:01:38.274ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11042025-05-08T22:01:38.275ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11052025-05-08T22:01:38.275ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 9.2968ms iteration = 7 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
11062025-05-08T22:01:38.277ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
11072025-05-08T22:01:38.277ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
11082025-05-08T22:01:38.277ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
11092025-05-08T22:01:38.278ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1074, tv_nsec: 20543038 } })
11102025-05-08T22:01:38.278ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163
11112025-05-08T22:01:38.279ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
11122025-05-08T22:01:38.279ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
11132025-05-08T22:01:38.279ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
11142025-05-08T22:01:38.279ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 19 reason = Timeout
11152025-05-08T22:01:38.279ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
11162025-05-08T22:01:38.279ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 7 reason = Timeout
11172025-05-08T22:01:38.279ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
11182025-05-08T22:01:38.279ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 7 reason = Timeout
11192025-05-08T22:01:38.279ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
11202025-05-08T22:01:38.279ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
11212025-05-08T22:01:38.279ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
11222025-05-08T22:01:38.279ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 7 reason = Timeout
11232025-05-08T22:01:38.279ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
11242025-05-08T22:01:38.279ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
11252025-05-08T22:01:38.279ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 7 reason = Timeout
11262025-05-08T22:01:38.279ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 8 reason = Timeout
11272025-05-08T22:01:38.279ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11282025-05-08T22:01:38.279ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
11292025-05-08T22:01:38.279ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 4 reason = Timeout
11302025-05-08T22:01:38.279ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
11312025-05-08T22:01:38.279ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 4 reason = Timeout
11322025-05-08T22:01:38.279ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:53
11332025-05-08T22:01:38.279ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 7 reason = Timeout
11342025-05-08T22:01:38.279ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
11352025-05-08T22:01:38.279ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 7 reason = Timeout
11362025-05-08T22:01:38.279ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
11372025-05-08T22:01:38.279ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
11382025-05-08T22:01:38.279ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 7 reason = Timeout
11392025-05-08T22:01:38.279ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
11402025-05-08T22:01:38.279ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
11412025-05-08T22:01:38.279ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
11422025-05-08T22:01:38.279ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
11432025-05-08T22:01:38.279ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
11442025-05-08T22:01:38.279ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
11452025-05-08T22:01:38.279ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
11462025-05-08T22:01:38.279ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11472025-05-08T22:01:38.279ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
11482025-05-08T22:01:38.279ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
11492025-05-08T22:01:38.279ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
11502025-05-08T22:01:38.279ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
11512025-05-08T22:01:38.279ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
11522025-05-08T22:01:38.281ZDEBGnexus (BackgroundTasks): activating background_task = webhook_deliverator iteration = 4 reason = Timeout
11532025-05-08T22:01:38.281ZDEBGnexus (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} }
11542025-05-08T22:01:38.281ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 4 reason = Timeout
11552025-05-08T22:01:38.281ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
11562025-05-08T22:01:38.281ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 7 reason = Timeout
11572025-05-08T22:01:38.281ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
11582025-05-08T22:01:38.281ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 7 reason = Timeout
11592025-05-08T22:01:38.281ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
11602025-05-08T22:01:38.281ZDEBGnexus (BackgroundTasks): activating background_task = webhook_dispatcher iteration = 4 reason = Timeout
11612025-05-08T22:01:38.281ZDEBGnexus (BackgroundTasks): roles background_task = webhook_dispatcher roles = RoleSet { roles: {} }
11622025-05-08T22:01:38.281ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 4 reason = Timeout
11632025-05-08T22:01:38.281ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
11642025-05-08T22:01:38.281ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 7 reason = Timeout
11652025-05-08T22:01:38.281ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
11662025-05-08T22:01:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
11672025-05-08T22:01:38.282ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 7 reason = Timeout
11682025-05-08T22:01:38.282ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
11692025-05-08T22:01:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database result = Ok(())
11702025-05-08T22:01:38.282ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 7 reason = Timeout
11712025-05-08T22:01:38.282ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
11722025-05-08T22:01:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
11732025-05-08T22:01:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
11742025-05-08T22:01:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
11752025-05-08T22:01:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
11762025-05-08T22:01:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database result = Ok(())
11772025-05-08T22:01:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
11782025-05-08T22:01:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
11792025-05-08T22:01:38.287ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11802025-05-08T22:01:38.288ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
11812025-05-08T22:01:38.288ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
11822025-05-08T22:01:38.288ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
11832025-05-08T22:01:38.288ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
11842025-05-08T22:01:38.288ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
11852025-05-08T22:01:38.288ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11862025-05-08T22:01:38.288ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11872025-05-08T22:01:38.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
11882025-05-08T22:01:38.288ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
11892025-05-08T22:01:38.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
11902025-05-08T22:01:38.289ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
11912025-05-08T22:01:38.291ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
11922025-05-08T22:01:38.291ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
11932025-05-08T22:01:38.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
11942025-05-08T22:01:38.292ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11952025-05-08T22:01:38.292ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
11962025-05-08T22:01:38.292ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
11972025-05-08T22:01:38.292ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
11982025-05-08T22:01:38.292ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
11992025-05-08T22:01:38.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
12002025-05-08T22:01:38.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
12012025-05-08T22:01:38.294ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
12022025-05-08T22:01:38.294ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
12032025-05-08T22:01:38.295ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
12042025-05-08T22:01:38.295ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12052025-05-08T22:01:38.298ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
12062025-05-08T22:01:38.298ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
12072025-05-08T22:01:38.298ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
12082025-05-08T22:01:38.300ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12092025-05-08T22:01:38.303ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
12102025-05-08T22:01:38.303ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
12112025-05-08T22:01:38.304ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
12122025-05-08T22:01:38.308ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12132025-05-08T22:01:38.309ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12142025-05-08T22:01:38.313ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
12152025-05-08T22:01:38.313ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
12162025-05-08T22:01:38.313ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
12172025-05-08T22:01:38.313ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
12182025-05-08T22:01:38.313ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
12192025-05-08T22:01:38.314ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
12202025-05-08T22:01:38.315ZDEBGnexus (BackgroundTasks): activation complete background_task = webhook_deliverator elapsed = 33.959954ms iteration = 4 status = {"by_rx":{},"error":null}
12212025-05-08T22:01:38.316ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 35.157187ms iteration = 4 status = {"errors":[],"found_port_ok":[]}
12222025-05-08T22:01:38.319ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
12232025-05-08T22:01:38.319ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
12242025-05-08T22:01:38.319ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 37.570972ms iteration = 4 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
12252025-05-08T22:01:38.326ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
12262025-05-08T22:01:38.326ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
12272025-05-08T22:01:38.340ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12282025-05-08T22:01:38.345ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
12292025-05-08T22:01:38.345ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12302025-05-08T22:01:38.346ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12312025-05-08T22:01:38.348ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12322025-05-08T22:01:38.348ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12332025-05-08T22:01:38.352ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
12342025-05-08T22:01:38.352ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
12352025-05-08T22:01:38.352ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
12362025-05-08T22:01:38.353ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
12372025-05-08T22:01:38.353ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
12382025-05-08T22:01:38.353ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
12392025-05-08T22:01:38.357ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12402025-05-08T22:01:38.357ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
12412025-05-08T22:01:38.357ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
12422025-05-08T22:01:38.357ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
12432025-05-08T22:01:38.358ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
12442025-05-08T22:01:38.358ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
12452025-05-08T22:01:38.358ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
12462025-05-08T22:01:38.358ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
12472025-05-08T22:01:38.359ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
12482025-05-08T22:01:38.359ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12492025-05-08T22:01:38.360ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
12502025-05-08T22:01:38.361ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
12512025-05-08T22:01:38.361ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
12522025-05-08T22:01:38.361ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
12532025-05-08T22:01:38.361ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
12542025-05-08T22:01:38.361ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
12552025-05-08T22:01:38.361ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
12562025-05-08T22:01:38.361ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
12572025-05-08T22:01:38.361ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
12582025-05-08T22:01:38.361ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
12592025-05-08T22:01:38.363ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
12602025-05-08T22:01:38.363ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
12612025-05-08T22:01:38.363ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
12622025-05-08T22:01:38.364ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
12632025-05-08T22:01:38.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
12642025-05-08T22:01:38.365ZDEBGnexus (BackgroundTasks): roles background_task = webhook_dispatcher roles = RoleSet { roles: {} }
12652025-05-08T22:01:38.366ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database result = Ok(())
12662025-05-08T22:01:38.369ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
12672025-05-08T22:01:38.369ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:137 found = 0 not_ready_to_be_deleted = 0
12682025-05-08T22:01:38.369ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 90.422327ms iteration = 4 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
12692025-05-08T22:01:38.373ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12702025-05-08T22:01:38.374ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
12712025-05-08T22:01:38.376ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1074, tv_nsec: 118555227 } })
12722025-05-08T22:01:38.376ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
12732025-05-08T22:01:38.376ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
12742025-05-08T22:01:38.376ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
12752025-05-08T22:01:38.376ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
12762025-05-08T22:01:38.377ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
12772025-05-08T22:01:38.377ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
12782025-05-08T22:01:38.381ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
12792025-05-08T22:01:38.381ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 114.606905ms iteration = 19 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
12802025-05-08T22:01:38.387ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
12812025-05-08T22:01:38.387ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
12822025-05-08T22:01:38.392ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12832025-05-08T22:01:38.396ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
12842025-05-08T22:01:38.396ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
12852025-05-08T22:01:38.397ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
12862025-05-08T22:01:38.397ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
12872025-05-08T22:01:38.397ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
12882025-05-08T22:01:38.397ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
12892025-05-08T22:01:38.397ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
12902025-05-08T22:01:38.399ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 117.580648ms iteration = 7 status = {"errors":[],"requests_created_ok":[]}
12912025-05-08T22:01:38.403ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
12922025-05-08T22:01:38.403ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
12932025-05-08T22:01:38.404ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d37c311-bf5c-4717-a37e-3a66941d6453", "content-length": "26", "date": "Thu, 08 May 2025 22:01:38 GMT"} })
12942025-05-08T22:01:38.404ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
12952025-05-08T22:01:38.405ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
12962025-05-08T22:01:38.405ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
12972025-05-08T22:01:38.406ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
12982025-05-08T22:01:38.406ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12992025-05-08T22:01:38.411ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
13002025-05-08T22:01:38.411ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
13012025-05-08T22:01:38.411ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
13022025-05-08T22:01:38.420ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
13032025-05-08T22:01:38.420ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
13042025-05-08T22:01:38.420ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
13052025-05-08T22:01:38.420ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
13062025-05-08T22:01:38.421ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
13072025-05-08T22:01:38.424ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:552 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
13082025-05-08T22:01:38.424ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 145.075411ms iteration = 7 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
13092025-05-08T22:01:38.432ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
13102025-05-08T22:01:38.432ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
13112025-05-08T22:01:38.433ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "576e710f-3184-4b59-bc6a-caddef6a2e32", "content-length": "26", "date": "Thu, 08 May 2025 22:01:38 GMT"} })
13122025-05-08T22:01:38.433ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
13132025-05-08T22:01:38.436ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
13142025-05-08T22:01:38.437ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 158.173011ms iteration = 19 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
13152025-05-08T22:01:38.448ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 166.764343ms iteration = 7 status = {"errors":[],"garbage_collect_requested":[]}
13162025-05-08T22:01:38.498ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
13172025-05-08T22:01:38.499ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
13182025-05-08T22:01:38.500ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 218.271634ms iteration = 7 status = {"errors":[],"finish_invoked_ok":[]}
13192025-05-08T22:01:38.625ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
13202025-05-08T22:01:38.625ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
13212025-05-08T22:01:38.691ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
13222025-05-08T22:01:38.696ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e zone_address = fd00:1122:3344:101::2
13232025-05-08T22:01:38.696ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
13242025-05-08T22:01:38.696ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2718a23-70c7-4123-802f-f9f9879f5d46", "content-length": "26", "date": "Thu, 08 May 2025 22:01:38 GMT"} })
13252025-05-08T22:01:38.696ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1174 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
13262025-05-08T22:01:38.698ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
13272025-05-08T22:01:38.749ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
13282025-05-08T22:01:38.749ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
13292025-05-08T22:01:38.749ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf369381-3efb-40b6-9137-bb6a0167f2fb", "content-length": "372", "date": "Thu, 08 May 2025 22:01:38 GMT"} })
13302025-05-08T22:01:38.749ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
13312025-05-08T22:01:38.749ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
13322025-05-08T22:01:38.752ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13332025-05-08T22:01:38.753ZDEBGnexus (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 485.23398ms iteration = 3 status = {"nsuccess":5}
13342025-05-08T22:01:38.754ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13352025-05-08T22:01:38.754ZDEBGnexus (BackgroundTasks): activation complete background_task = webhook_dispatcher elapsed = 473.043839ms iteration = 4 status = {"dispatched":[],"errors":[],"glob_version":"140.0.0","globs_reprocessed":{},"no_receivers":[]}
13362025-05-08T22:01:38.755ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13372025-05-08T22:01:38.756ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
13382025-05-08T22:01:38.756ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
13392025-05-08T22:01:38.756ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
13402025-05-08T22:01:38.757ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
13412025-05-08T22:01:38.757ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
13422025-05-08T22:01:38.758ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
13432025-05-08T22:01:38.758ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
13442025-05-08T22:01:38.758ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
13452025-05-08T22:01:38.758ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
13462025-05-08T22:01:38.760ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 480.823723ms iteration = 7 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
13472025-05-08T22:01:38.762ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
13482025-05-08T22:01:38.763ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
13492025-05-08T22:01:38.764ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 500.009488ms iteration = 7 status = null
13502025-05-08T22:01:38.766ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13512025-05-08T22:01:38.772ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 2f2f2c32-8d58-4801-a669-9618eac0c3f7, lookup_type: ById(2f2f2c32-8d58-4801-a669-9618eac0c3f7) } result = Ok(())
13522025-05-08T22:01:38.772ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
13532025-05-08T22:01:38.772ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
13542025-05-08T22:01:38.775ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 496.42409ms iteration = 7 status = 0
13552025-05-08T22:01:38.803ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
13562025-05-08T22:01:38.804ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "fed6cbbb-e64e-4788-8fd2-9658bbe5b161", "date": "Thu, 08 May 2025 22:01:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
13572025-05-08T22:01:38.804ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
13582025-05-08T22:01:38.806ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "61eaf9c3-643d-4974-b3a1-61d7632f59cf", "date": "Thu, 08 May 2025 22:01:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
13592025-05-08T22:01:38.806ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
13602025-05-08T22:01:38.807ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
13612025-05-08T22:01:38.807ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 543.281281ms iteration = 6 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
13622025-05-08T22:01:38.810ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 546.542676ms iteration = 7 status = {}
13632025-05-08T22:01:38.815ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13642025-05-08T22:01:38.816ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
13652025-05-08T22:01:38.817ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
13662025-05-08T22:01:38.817ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
13672025-05-08T22:01:38.817ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
13682025-05-08T22:01:38.817ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 537.827092ms iteration = 7 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
13692025-05-08T22:01:38.817ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
13702025-05-08T22:01:38.817ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
13712025-05-08T22:01:38.818ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
13722025-05-08T22:01:38.819ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
13732025-05-08T22:01:38.820ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
13742025-05-08T22:01:38.821ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
13752025-05-08T22:01:38.821ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = d8128696-db5a-4b88-914a-69b28494e50e dns_zone_name = oxide.test found_so_far = 0 version = 2
13762025-05-08T22:01:38.821ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
13772025-05-08T22:01:38.821ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
13782025-05-08T22:01:38.821ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13792025-05-08T22:01:38.821ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 539.567411ms iteration = 7 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
13802025-05-08T22:01:38.821ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:362
13812025-05-08T22:01:38.822ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 553.668652ms iteration = 7 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
13822025-05-08T22:01:38.823ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
13832025-05-08T22:01:38.823ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
13842025-05-08T22:01:38.823ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
13852025-05-08T22:01:38.825ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
13862025-05-08T22:01:38.825ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "3a94f9ad-b318-4a57-9fce-f3ceaa70c91a", "date": "Thu, 08 May 2025 22:01:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
13872025-05-08T22:01:38.825ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
13882025-05-08T22:01:38.825ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
13892025-05-08T22:01:38.825ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 7586992a-6fc5-4073-8962-728778f16aa5 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
13902025-05-08T22:01:38.825ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
13912025-05-08T22:01:38.826ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
13922025-05-08T22:01:38.827ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "5724f268-3523-4719-847b-f29c665b383d", "date": "Thu, 08 May 2025 22:01:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
13932025-05-08T22:01:38.827ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
13942025-05-08T22:01:38.827ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13952025-05-08T22:01:38.829ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "b72b77c7-84fb-4ee0-91ca-76e269bf1144", "date": "Thu, 08 May 2025 22:01:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
13962025-05-08T22:01:38.829ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
13972025-05-08T22:01:38.830ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
13982025-05-08T22:01:38.830ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
13992025-05-08T22:01:38.830ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
14002025-05-08T22:01:38.831ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
14012025-05-08T22:01:38.831ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 566.933413ms iteration = 6 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
14022025-05-08T22:01:38.832ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 7586992a-6fc5-4073-8962-728778f16aa5 dns_zone_name = control-plane.oxide.internal found_so_far = 55 version = 1
14032025-05-08T22:01:38.832ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
14042025-05-08T22:01:38.832ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-05-08 21:59:21.182334 UTC dns_group = internal generation = 1
14052025-05-08T22:01:38.832ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 568.310008ms iteration = 4 status = {"generation":1}
14062025-05-08T22:01:38.853ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
14072025-05-08T22:01:38.853ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
14082025-05-08T22:01:38.864ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14092025-05-08T22:01:38.872ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
14102025-05-08T22:01:38.873ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 591.307154ms iteration = 7 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
14112025-05-08T22:01:38.874ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
14122025-05-08T22:01:38.874ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
14132025-05-08T22:01:38.874ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
14142025-05-08T22:01:38.875ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 595.688762ms iteration = 8 status = {"physical_disks_added":0}
14152025-05-08T22:01:38.876ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = d8128696-db5a-4b88-914a-69b28494e50e dns_zone_name = oxide.test found_so_far = 1 version = 2
14162025-05-08T22:01:38.876ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
14172025-05-08T22:01:38.876ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-05-08 21:59:21.759368 UTC dns_group = external generation = 2
14182025-05-08T22:01:38.876ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 611.906124ms iteration = 4 status = {"generation":2}
14192025-05-08T22:01:38.879ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-05-08T21:58:35.560911Z, time_modified: 2025-05-08T21:58:35.560911Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, db_silo: Silo { identity: SiloIdentity { id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-05-08T21:59:21.709488Z, time_modified: 2025-05-08T21:59:21.709488Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, db_silo: Silo { identity: SiloIdentity { id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-05-08T21:59:21.709488Z, time_modified: 2025-05-08T21:59:21.709488Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47" }] }) }
14202025-05-08T22:01:38.879ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 615.045468ms iteration = 4 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"7137ee58-6aa7-4199-a248-feadf3b8bafb","tls_certs":["37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47"]}},"default_endpoint":{"silo_id":"7137ee58-6aa7-4199-a248-feadf3b8bafb","tls_certs":["37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
14212025-05-08T22:01:38.899ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14222025-05-08T22:01:38.899ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14232025-05-08T22:01:38.901ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: cd0b930c-534b-46ad-ac3b-2eaa94b0bf2c, rack_id: 64f3d6f4-89b8-48aa-a622-b684e261667e, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(65f83a36-18fb-49b7-9914-87c8517d200e) } rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
14242025-05-08T22:01:38.901ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14252025-05-08T22:01:38.902ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14262025-05-08T22:01:38.928ZINFOnexus (BackgroundTasks): instance reincarnation completed successfully background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:113 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0
14272025-05-08T22:01:38.928ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 649.342636ms iteration = 4 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
14282025-05-08T22:01:38.963ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "2b76ab89-e145-4ee0-bdad-7c455bb75197", "date": "Thu, 08 May 2025 22:01:38 GMT"} })
14292025-05-08T22:01:38.963ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14302025-05-08T22:01:38.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14312025-05-08T22:01:38.964ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8620985-4682-4082-b467-2832c034ede8", "content-length": "691", "date": "Thu, 08 May 2025 22:01:38 GMT"} })
14322025-05-08T22:01:38.964ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
14332025-05-08T22:01:38.964ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
14342025-05-08T22:01:38.964ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
14352025-05-08T22:01:38.965ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 686.554074ms iteration = 7 status = {}
14362025-05-08T22:01:38.966ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14372025-05-08T22:01:38.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14382025-05-08T22:01:38.970ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14392025-05-08T22:01:38.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14402025-05-08T22:01:38.974ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
14412025-05-08T22:01:38.974ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
14422025-05-08T22:01:38.974ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "92399f88-2777-4206-aa71-bd990cbb34ae", "date": "Thu, 08 May 2025 22:01:38 GMT"} })
14432025-05-08T22:01:38.975ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 696.422749ms iteration = 7 status = {}
14442025-05-08T22:01:39.061ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1833 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
14452025-05-08T22:01:39.061ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1860 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 65f83a36-18fb-49b7-9914-87c8517d200e
14462025-05-08T22:01:39.061ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
14472025-05-08T22:01:39.061ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
14482025-05-08T22:01:39.061ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
14492025-05-08T22:01:39.061ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
14502025-05-08T22:01:39.061ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
14512025-05-08T22:01:39.061ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2007 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
14522025-05-08T22:01:39.062ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
14532025-05-08T22:01:39.062ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2043 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
14542025-05-08T22:01:39.062ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
14552025-05-08T22:01:39.062ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14562025-05-08T22:01:39.062ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14572025-05-08T22:01:39.064ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
14582025-05-08T22:01:39.064ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
14592025-05-08T22:01:39.172ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "4102cede-5e50-4af7-b3bb-bbc9c221cf9d", "date": "Thu, 08 May 2025 22:01:38 GMT"} })
14602025-05-08T22:01:39.172ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch = Switch0
14612025-05-08T22:01:39.172ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
14622025-05-08T22:01:39.173ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d16fb12-fbf3-4c24-ae09-d25701751eff", "content-length": "682", "date": "Thu, 08 May 2025 22:01:38 GMT"} })
14632025-05-08T22:01:39.173ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14642025-05-08T22:01:39.173ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14652025-05-08T22:01:39.177ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14662025-05-08T22:01:39.177ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14672025-05-08T22:01:39.180ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14682025-05-08T22:01:39.180ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14692025-05-08T22:01:39.182ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14702025-05-08T22:01:39.182ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14712025-05-08T22:01:39.188ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 909.442378ms iteration = 7 status = {}
14722025-05-08T22:01:41.934ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 434 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 70719608-5448-4126-b20f-91c639ddec52 response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
14732025-05-08T22:01:48.266ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 20 reason = Timeout
14742025-05-08T22:01:48.266ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
14752025-05-08T22:01:48.267ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
14762025-05-08T22:01:48.268ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14772025-05-08T22:01:48.271ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
14782025-05-08T22:01:48.271ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
14792025-05-08T22:01:48.271ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
14802025-05-08T22:01:48.273ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
14812025-05-08T22:01:48.274ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
14822025-05-08T22:01:48.275ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14832025-05-08T22:01:48.278ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
14842025-05-08T22:01:48.278ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
14852025-05-08T22:01:48.278ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
14862025-05-08T22:01:48.279ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 20 reason = Timeout
14872025-05-08T22:01:48.279ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
14882025-05-08T22:01:48.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
14892025-05-08T22:01:48.288ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
14902025-05-08T22:01:48.288ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.122727ms iteration = 20 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
14912025-05-08T22:01:48.316ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
14922025-05-08T22:01:48.317ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
14932025-05-08T22:01:48.348ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 68.773485ms iteration = 20 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
14942025-05-08T22:01:51.927ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::d]:44638
14952025-05-08T22:01:51.927ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44638 req_id = 114d526e-9091-44ae-b14c-cb72d92d92d1 roles = RoleSet { roles: {} } uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?sort_by=id_ascending
14962025-05-08T22:01:51.928ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44638 req_id = 114d526e-9091-44ae-b14c-cb72d92d92d1 resource = Database result = Ok(()) uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?sort_by=id_ascending
14972025-05-08T22:01:51.931ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3888 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44638 req_id = 114d526e-9091-44ae-b14c-cb72d92d92d1 response_code = 200 uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?sort_by=id_ascending
14982025-05-08T22:01:51.934ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44638 req_id = c0038eb4-0088-413c-90af-a2e84f4922d5 roles = RoleSet { roles: {} } uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZGQ4MjYzNC0yODQ0LTRmOTctOTc0Ni0wNzA5OTA0NGMxOWQifX0%3D
14992025-05-08T22:01:51.934ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44638 req_id = c0038eb4-0088-413c-90af-a2e84f4922d5 resource = Database result = Ok(()) uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZGQ4MjYzNC0yODQ0LTRmOTctOTc0Ni0wNzA5OTA0NGMxOWQifX0%3D
15002025-05-08T22:01:51.935ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 381 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = c7768ba3-4925-49c0-8fbe-3652c682e944 response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
15012025-05-08T22:01:51.936ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 5266 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44638 req_id = c0038eb4-0088-413c-90af-a2e84f4922d5 response_code = 200 uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZGQ4MjYzNC0yODQ0LTRmOTctOTc0Ni0wNzA5OTA0NGMxOWQifX0%3D
15022025-05-08T22:01:58.267ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 21 reason = Timeout
15032025-05-08T22:01:58.267ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15042025-05-08T22:01:58.267ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15052025-05-08T22:01:58.268ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15062025-05-08T22:01:58.271ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
15072025-05-08T22:01:58.271ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15082025-05-08T22:01:58.271ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15092025-05-08T22:01:58.273ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15102025-05-08T22:01:58.273ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15112025-05-08T22:01:58.275ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15122025-05-08T22:01:58.278ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
15132025-05-08T22:01:58.278ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15142025-05-08T22:01:58.278ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15152025-05-08T22:01:58.280ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 21 reason = Timeout
15162025-05-08T22:01:58.280ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
15172025-05-08T22:01:58.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
15182025-05-08T22:01:58.282ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
15192025-05-08T22:01:58.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
15202025-05-08T22:01:58.283ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.632239ms iteration = 21 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
15212025-05-08T22:01:58.288ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
15222025-05-08T22:01:58.288ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.788103ms iteration = 21 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
15232025-05-08T22:02:01.935ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 283 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 65d45d3a-a158-4ec2-86bd-ec9c8eec29a5 response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
15242025-05-08T22:02:06.927ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44638 req_id = b2932e28-8424-4997-990f-450cb25fd2eb roles = RoleSet { roles: {} } uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?sort_by=id_ascending
15252025-05-08T22:02:06.928ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44638 req_id = b2932e28-8424-4997-990f-450cb25fd2eb resource = Database result = Ok(()) uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?sort_by=id_ascending
15262025-05-08T22:02:06.935ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 7408 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44638 req_id = b2932e28-8424-4997-990f-450cb25fd2eb response_code = 200 uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?sort_by=id_ascending
15272025-05-08T22:02:06.935ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44638 req_id = 421ea530-b202-4ff2-a200-88b36eb54465 roles = RoleSet { roles: {} } uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZGQ4MjYzNC0yODQ0LTRmOTctOTc0Ni0wNzA5OTA0NGMxOWQifX0%3D
15282025-05-08T22:02:06.935ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44638 req_id = 421ea530-b202-4ff2-a200-88b36eb54465 resource = Database result = Ok(()) uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZGQ4MjYzNC0yODQ0LTRmOTctOTc0Ni0wNzA5OTA0NGMxOWQifX0%3D
15292025-05-08T22:02:06.939ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3685 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:44638 req_id = 421ea530-b202-4ff2-a200-88b36eb54465 response_code = 200 uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZGQ4MjYzNC0yODQ0LTRmOTctOTc0Ni0wNzA5OTA0NGMxOWQifX0%3D
15302025-05-08T22:02:08.265ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 8 reason = Timeout
15312025-05-08T22:02:08.265ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
15322025-05-08T22:02:08.265ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 8 reason = Timeout
15332025-05-08T22:02:08.265ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
15342025-05-08T22:02:08.265ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
15352025-05-08T22:02:08.265ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
15362025-05-08T22:02:08.266ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 22 reason = Timeout
15372025-05-08T22:02:08.266ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 8 reason = Timeout
15382025-05-08T22:02:08.266ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15392025-05-08T22:02:08.267ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15402025-05-08T22:02:08.268ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1104, tv_nsec: 10577322 } })
15412025-05-08T22:02:08.268ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1104, tv_nsec: 10675673 } })
15422025-05-08T22:02:08.268ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 8 reason = Timeout
15432025-05-08T22:02:08.268ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
15442025-05-08T22:02:08.268ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163
15452025-05-08T22:02:08.268ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
15462025-05-08T22:02:08.268ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1163
15472025-05-08T22:02:08.269ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15482025-05-08T22:02:08.270ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.424827ms iteration = 8 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
15492025-05-08T22:02:08.270ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15502025-05-08T22:02:08.272ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
15512025-05-08T22:02:08.272ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15522025-05-08T22:02:08.272ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15532025-05-08T22:02:08.273ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
15542025-05-08T22:02:08.273ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
15552025-05-08T22:02:08.273ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
15562025-05-08T22:02:08.274ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15572025-05-08T22:02:08.274ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15582025-05-08T22:02:08.275ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15592025-05-08T22:02:08.278ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
15602025-05-08T22:02:08.278ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15612025-05-08T22:02:08.278ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15622025-05-08T22:02:08.279ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 22 reason = Timeout
15632025-05-08T22:02:08.279ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
15642025-05-08T22:02:08.279ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 8 reason = Timeout
15652025-05-08T22:02:08.279ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
15662025-05-08T22:02:08.279ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
15672025-05-08T22:02:08.279ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 8 reason = Timeout
15682025-05-08T22:02:08.279ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
15692025-05-08T22:02:08.279ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 8 reason = Timeout
15702025-05-08T22:02:08.279ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
15712025-05-08T22:02:08.279ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
15722025-05-08T22:02:08.279ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 8 reason = Timeout
15732025-05-08T22:02:08.279ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 8 reason = Timeout
15742025-05-08T22:02:08.279ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
15752025-05-08T22:02:08.279ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 9 reason = Timeout
15762025-05-08T22:02:08.279ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
15772025-05-08T22:02:08.279ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
15782025-05-08T22:02:08.279ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 8 reason = Timeout
15792025-05-08T22:02:08.280ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 8 reason = Timeout
15802025-05-08T22:02:08.280ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
15812025-05-08T22:02:08.280ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
15822025-05-08T22:02:08.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
15832025-05-08T22:02:08.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
15842025-05-08T22:02:08.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
15852025-05-08T22:02:08.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
15862025-05-08T22:02:08.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
15872025-05-08T22:02:08.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
15882025-05-08T22:02:08.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
15892025-05-08T22:02:08.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
15902025-05-08T22:02:08.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
15912025-05-08T22:02:08.282ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 8 reason = Timeout
15922025-05-08T22:02:08.282ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 8 reason = Timeout
15932025-05-08T22:02:08.282ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
15942025-05-08T22:02:08.282ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
15952025-05-08T22:02:08.282ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 8 reason = Timeout
15962025-05-08T22:02:08.282ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
15972025-05-08T22:02:08.282ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 8 reason = Timeout
15982025-05-08T22:02:08.282ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
15992025-05-08T22:02:08.282ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 8 reason = Timeout
16002025-05-08T22:02:08.282ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
16012025-05-08T22:02:08.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
16022025-05-08T22:02:08.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
16032025-05-08T22:02:08.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
16042025-05-08T22:02:08.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
16052025-05-08T22:02:08.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
16062025-05-08T22:02:08.287ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
16072025-05-08T22:02:08.287ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
16082025-05-08T22:02:08.287ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
16092025-05-08T22:02:08.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
16102025-05-08T22:02:08.300ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
16112025-05-08T22:02:08.300ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 33.373157ms iteration = 22 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
16122025-05-08T22:02:08.317ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
16132025-05-08T22:02:08.317ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
16142025-05-08T22:02:08.318ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "a548eb71-2fc8-4b8a-bbb9-9c98fc9b4d8e", "content-length": "26", "date": "Thu, 08 May 2025 22:02:07 GMT"} })
16152025-05-08T22:02:08.318ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
16162025-05-08T22:02:08.319ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
16172025-05-08T22:02:08.319ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
16182025-05-08T22:02:08.320ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae5ffc37-6f63-475e-b4cd-5cbd264d121c", "content-length": "26", "date": "Thu, 08 May 2025 22:02:07 GMT"} })
16192025-05-08T22:02:08.320ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
16202025-05-08T22:02:08.320ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
16212025-05-08T22:02:08.321ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
16222025-05-08T22:02:08.341ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
16232025-05-08T22:02:08.341ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
16242025-05-08T22:02:08.342ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
16252025-05-08T22:02:08.343ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
16262025-05-08T22:02:08.343ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16272025-05-08T22:02:08.343ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16282025-05-08T22:02:08.345ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 65.332548ms iteration = 22 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
16292025-05-08T22:02:08.345ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
16302025-05-08T22:02:08.346ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
16312025-05-08T22:02:08.347ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
16322025-05-08T22:02:08.347ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
16332025-05-08T22:02:08.347ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
16342025-05-08T22:02:08.348ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
16352025-05-08T22:02:08.348ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
16362025-05-08T22:02:08.348ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
16372025-05-08T22:02:08.351ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16382025-05-08T22:02:08.355ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 72.804298ms iteration = 8 status = {"errors":[],"finish_invoked_ok":[]}
16392025-05-08T22:02:08.355ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1104, tv_nsec: 97268809 } })
16402025-05-08T22:02:08.355ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
16412025-05-08T22:02:08.355ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
16422025-05-08T22:02:08.355ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
16432025-05-08T22:02:08.355ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
16442025-05-08T22:02:08.355ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
16452025-05-08T22:02:08.355ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
16462025-05-08T22:02:08.355ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
16472025-05-08T22:02:08.356ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
16482025-05-08T22:02:08.362ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16492025-05-08T22:02:08.362ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16502025-05-08T22:02:08.365ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16512025-05-08T22:02:08.365ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
16522025-05-08T22:02:08.365ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
16532025-05-08T22:02:08.366ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 83.861227ms iteration = 8 status = {"errors":[],"garbage_collect_requested":[]}
16542025-05-08T22:02:08.366ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
16552025-05-08T22:02:08.367ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
16562025-05-08T22:02:08.367ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
16572025-05-08T22:02:08.367ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
16582025-05-08T22:02:08.368ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
16592025-05-08T22:02:08.368ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
16602025-05-08T22:02:08.368ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
16612025-05-08T22:02:08.368ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
16622025-05-08T22:02:08.368ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
16632025-05-08T22:02:08.369ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 87.124972ms iteration = 8 status = {"errors":[],"requests_created_ok":[]}
16642025-05-08T22:02:08.371ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
16652025-05-08T22:02:08.371ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
16662025-05-08T22:02:08.372ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
16672025-05-08T22:02:08.372ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
16682025-05-08T22:02:08.373ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
16692025-05-08T22:02:08.373ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
16702025-05-08T22:02:08.374ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:552 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
16712025-05-08T22:02:08.374ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 94.4496ms iteration = 8 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
16722025-05-08T22:02:08.374ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16732025-05-08T22:02:08.377ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
16742025-05-08T22:02:08.377ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
16752025-05-08T22:02:08.377ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
16762025-05-08T22:02:08.378ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 96.508001ms iteration = 8 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
16772025-05-08T22:02:08.397ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:362
16782025-05-08T22:02:08.397ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 129.145661ms iteration = 8 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
16792025-05-08T22:02:08.399ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
16802025-05-08T22:02:08.399ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
16812025-05-08T22:02:08.400ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
16822025-05-08T22:02:08.400ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
16832025-05-08T22:02:08.400ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 120.59347ms iteration = 8 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
16842025-05-08T22:02:08.403ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
16852025-05-08T22:02:08.403ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
16862025-05-08T22:02:08.404ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 124.935736ms iteration = 8 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
16872025-05-08T22:02:08.406ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 141.2089ms iteration = 8 status = null
16882025-05-08T22:02:08.406ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16892025-05-08T22:02:08.410ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
16902025-05-08T22:02:08.410ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
16912025-05-08T22:02:08.410ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
16922025-05-08T22:02:08.413ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 148.388087ms iteration = 8 status = {}
16932025-05-08T22:02:08.424ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e zone_address = fd00:1122:3344:101::2
16942025-05-08T22:02:08.424ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
16952025-05-08T22:02:08.424ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "533220d2-3a00-4f79-b23c-14f685833159", "content-length": "26", "date": "Thu, 08 May 2025 22:02:08 GMT"} })
16962025-05-08T22:02:08.424ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1174 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
16972025-05-08T22:02:08.425ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
16982025-05-08T22:02:08.448ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 166.215407ms iteration = 8 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
16992025-05-08T22:02:08.458ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
17002025-05-08T22:02:08.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
17012025-05-08T22:02:08.498ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
17022025-05-08T22:02:08.498ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 218.81296ms iteration = 9 status = {"physical_disks_added":0}
17032025-05-08T22:02:08.498ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
17042025-05-08T22:02:08.547ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
17052025-05-08T22:02:08.547ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
17062025-05-08T22:02:08.553ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: cd0b930c-534b-46ad-ac3b-2eaa94b0bf2c, rack_id: 64f3d6f4-89b8-48aa-a622-b684e261667e, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(65f83a36-18fb-49b7-9914-87c8517d200e) } rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
17072025-05-08T22:02:08.553ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
17082025-05-08T22:02:08.553ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
17092025-05-08T22:02:08.572ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1833 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
17102025-05-08T22:02:08.572ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1860 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 65f83a36-18fb-49b7-9914-87c8517d200e
17112025-05-08T22:02:08.572ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
17122025-05-08T22:02:08.572ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
17132025-05-08T22:02:08.572ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
17142025-05-08T22:02:08.572ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
17152025-05-08T22:02:08.572ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
17162025-05-08T22:02:08.572ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2007 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
17172025-05-08T22:02:08.572ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
17182025-05-08T22:02:08.572ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2043 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
17192025-05-08T22:02:08.573ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
17202025-05-08T22:02:08.573ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
17212025-05-08T22:02:08.573ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
17222025-05-08T22:02:08.576ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
17232025-05-08T22:02:08.576ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
17242025-05-08T22:02:08.611ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
17252025-05-08T22:02:08.612ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
17262025-05-08T22:02:08.618ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17272025-05-08T22:02:08.620ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
17282025-05-08T22:02:08.620ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
17292025-05-08T22:02:08.621ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 2f2f2c32-8d58-4801-a669-9618eac0c3f7, lookup_type: ById(2f2f2c32-8d58-4801-a669-9618eac0c3f7) } result = Ok(())
17302025-05-08T22:02:08.621ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
17312025-05-08T22:02:08.622ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
17322025-05-08T22:02:08.623ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17332025-05-08T22:02:08.624ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 344.266772ms iteration = 8 status = 0
17342025-05-08T22:02:08.626ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
17352025-05-08T22:02:08.626ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
17362025-05-08T22:02:08.627ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
17372025-05-08T22:02:08.658ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
17382025-05-08T22:02:08.658ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
17392025-05-08T22:02:08.704ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17402025-05-08T22:02:08.707ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
17412025-05-08T22:02:08.707ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
17422025-05-08T22:02:08.708ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
17432025-05-08T22:02:08.722ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "ec6457c2-d9ac-4800-9049-46b3fb0a8546", "date": "Thu, 08 May 2025 22:02:08 GMT"} })
17442025-05-08T22:02:08.722ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch = Switch0
17452025-05-08T22:02:08.722ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
17462025-05-08T22:02:08.723ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "7eb60356-1442-41b6-a815-27d2b905e472", "content-length": "372", "date": "Thu, 08 May 2025 22:02:08 GMT"} })
17472025-05-08T22:02:08.723ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
17482025-05-08T22:02:08.723ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "887aeb6b-e8fd-4bbf-8a41-63da5120ae36", "content-length": "682", "date": "Thu, 08 May 2025 22:02:08 GMT"} })
17492025-05-08T22:02:08.723ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
17502025-05-08T22:02:08.724ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
17512025-05-08T22:02:08.724ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
17522025-05-08T22:02:08.729ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
17532025-05-08T22:02:08.729ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
17542025-05-08T22:02:08.798ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
17552025-05-08T22:02:08.798ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
17562025-05-08T22:02:08.803ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "c0f231c4-a639-4aa1-ab23-43a8d05f97c4", "date": "Thu, 08 May 2025 22:02:08 GMT"} })
17572025-05-08T22:02:08.803ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
17582025-05-08T22:02:08.804ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
17592025-05-08T22:02:08.807ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
17602025-05-08T22:02:08.807ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
17612025-05-08T22:02:08.811ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
17622025-05-08T22:02:08.812ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
17632025-05-08T22:02:08.816ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
17642025-05-08T22:02:08.816ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
17652025-05-08T22:02:08.822ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "a748c085-522a-43ee-b988-3e854c277b6d", "date": "Thu, 08 May 2025 22:02:08 GMT"} })
17662025-05-08T22:02:08.824ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 544.348733ms iteration = 8 status = {}
17672025-05-08T22:02:08.834ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
17682025-05-08T22:02:08.834ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
17692025-05-08T22:02:08.836ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
17702025-05-08T22:02:08.837ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
17712025-05-08T22:02:08.847ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 567.167136ms iteration = 8 status = {}
17722025-05-08T22:02:08.935ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
17732025-05-08T22:02:08.935ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "398c1fd7-ee84-46a5-ab2c-99f92f0f0e89", "content-length": "691", "date": "Thu, 08 May 2025 22:02:08 GMT"} })
17742025-05-08T22:02:08.935ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
17752025-05-08T22:02:08.935ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
17762025-05-08T22:02:08.935ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
17772025-05-08T22:02:08.936ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 656.455441ms iteration = 8 status = {}
17782025-05-08T22:02:09.014ZDEBGnexus (producer-registration-task): registering as oximeter producer with Nexus registration_count = 1
17792025-05-08T22:02:09.014ZDEBGnexus (producer-registration-task): will register with Nexus at [fd00:1122:3344:101::b]:12221
17802025-05-08T22:02:09.044ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::b]:12221/metrics/producers
17812025-05-08T22:02:09.044ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::b]:32993
17822025-05-08T22:02:09.044ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:32993 req_id = 2e361be0-8bb5-463f-9f01-2bd6548f6d8a roles = RoleSet { roles: {} } uri = /metrics/producers
17832025-05-08T22:02:09.044ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:32993 req_id = 2e361be0-8bb5-463f-9f01-2bd6548f6d8a resource = Database result = Ok(()) uri = /metrics/producers
17842025-05-08T22:02:09.114ZINFOnexus: assigned collector to new producer collector_id = 22ae79e2-3731-44c9-8346-78c5ddbfe577 file = nexus/src/app/oximeter.rs:76 producer_id = 85a39cf9-9419-458f-af2e-c8b40d8f5f79
17852025-05-08T22:02:09.114ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 70197 local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:32993 req_id = 2e361be0-8bb5-463f-9f01-2bd6548f6d8a response_code = 201 uri = /metrics/producers
17862025-05-08T22:02:09.114ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "2e361be0-8bb5-463f-9f01-2bd6548f6d8a", "content-length": "41", "date": "Thu, 08 May 2025 22:02:09 GMT"} })
17872025-05-08T22:02:09.115ZDEBGnexus (producer-registration-task): successfully registered with Nexus, pausing until time to renew lease lease_duration = 600s wait_period = 150s
17882025-05-08T22:02:11.935ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 293 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = adee6fc5-1aa0-4845-b209-1621e68fd74d response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
17892025-05-08T22:02:18.267ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 23 reason = Timeout
17902025-05-08T22:02:18.267ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
17912025-05-08T22:02:18.268ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17922025-05-08T22:02:18.269ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17932025-05-08T22:02:18.272ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
17942025-05-08T22:02:18.272ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
17952025-05-08T22:02:18.272ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17962025-05-08T22:02:18.274ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
17972025-05-08T22:02:18.275ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17982025-05-08T22:02:18.276ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17992025-05-08T22:02:18.279ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
18002025-05-08T22:02:18.279ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18012025-05-08T22:02:18.279ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
18022025-05-08T22:02:18.280ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 23 reason = Timeout
18032025-05-08T22:02:18.280ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
18042025-05-08T22:02:18.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
18052025-05-08T22:02:18.290ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
18062025-05-08T22:02:18.290ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.592072ms iteration = 23 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
18072025-05-08T22:02:18.308ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
18082025-05-08T22:02:18.309ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
18092025-05-08T22:02:18.340ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 60.151793ms iteration = 23 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
18102025-05-08T22:02:21.935ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 297 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = c26b56e5-9415-43a6-8f47-aa68b8a3edc3 response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
18112025-05-08T22:02:28.267ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 24 reason = Timeout
18122025-05-08T22:02:28.267ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18132025-05-08T22:02:28.268ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
18142025-05-08T22:02:28.269ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18152025-05-08T22:02:28.272ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
18162025-05-08T22:02:28.272ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18172025-05-08T22:02:28.272ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
18182025-05-08T22:02:28.274ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18192025-05-08T22:02:28.274ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
18202025-05-08T22:02:28.277ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18212025-05-08T22:02:28.279ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
18222025-05-08T22:02:28.280ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18232025-05-08T22:02:28.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
18242025-05-08T22:02:28.280ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 24 reason = Timeout
18252025-05-08T22:02:28.280ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
18262025-05-08T22:02:28.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
18272025-05-08T22:02:28.282ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
18282025-05-08T22:02:28.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
18292025-05-08T22:02:28.284ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.283026ms iteration = 24 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
18302025-05-08T22:02:28.289ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
18312025-05-08T22:02:28.289ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.812833ms iteration = 24 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
18322025-05-08T22:02:31.934ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 247 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 9355307a-2795-42f9-a0b0-eafbf8514f6f response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
18332025-05-08T22:02:38.265ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 9 reason = Timeout
18342025-05-08T22:02:38.265ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 7 reason = Timeout
18352025-05-08T22:02:38.265ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
18362025-05-08T22:02:38.265ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 5 reason = Timeout
18372025-05-08T22:02:38.265ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 5 reason = Timeout
18382025-05-08T22:02:38.265ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
18392025-05-08T22:02:38.265ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-05-08 21:52:38.265415839 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
18402025-05-08T22:02:38.265ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
18412025-05-08T22:02:38.265ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 5 reason = Timeout
18422025-05-08T22:02:38.265ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 7 reason = Timeout
18432025-05-08T22:02:38.265ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 5 reason = Timeout
18442025-05-08T22:02:38.265ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
18452025-05-08T22:02:38.265ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 9 reason = Timeout
18462025-05-08T22:02:38.265ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
18472025-05-08T22:02:38.265ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
18482025-05-08T22:02:38.265ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 5 reason = Timeout
18492025-05-08T22:02:38.265ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 5 reason = Timeout
18502025-05-08T22:02:38.265ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
18512025-05-08T22:02:38.265ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
18522025-05-08T22:02:38.266ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
18532025-05-08T22:02:38.266ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
18542025-05-08T22:02:38.266ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
18552025-05-08T22:02:38.266ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
18562025-05-08T22:02:38.266ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
18572025-05-08T22:02:38.266ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
18582025-05-08T22:02:38.266ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("0ccabf41-81cd-45e7-bb26-335edaa2ee47.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("38a08dd6-6a75-4558-b4f8-df853f4d03d9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("38a08dd6-6a75-4558-b4f8-df853f4d03d9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::9))) }], valid_until: Instant { tv_sec: 1134, tv_nsec: 7990214 } })
18592025-05-08T22:02:38.266ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("228ed1c1-3cc1-4055-aae9-5595ad856b86.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("8b245480-f91d-406b-bb08-850e1c3e3fc5.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c8f20f67-ea63-4dbc-b0f1-c30876daff82.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c8f20f67-ea63-4dbc-b0f1-c30876daff82.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 1134, tv_nsec: 8131316 } })
18602025-05-08T22:02:38.266ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
18612025-05-08T22:02:38.266ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } dns_group = external
18622025-05-08T22:02:38.266ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.307674ms iteration = 5 status = {"addresses":["[fd00:1122:3344:101::8]:5353","[fd00:1122:3344:101::9]:5353"]}
18632025-05-08T22:02:38.266ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
18642025-05-08T22:02:38.266ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 25 reason = Timeout
18652025-05-08T22:02:38.266ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18662025-05-08T22:02:38.266ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 6 reason = Timeout
18672025-05-08T22:02:38.266ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:85 reason = blueprint disabled target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2
18682025-05-08T22:02:38.266ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 9\u{b5}s iteration = 6 status = {"enabled":false,"target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2"}
18692025-05-08T22:02:38.266ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 9 reason = Timeout
18702025-05-08T22:02:38.267ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
18712025-05-08T22:02:38.267ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.508256ms iteration = 5 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:1::1]:5353"]}
18722025-05-08T22:02:38.267ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
18732025-05-08T22:02:38.267ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
18742025-05-08T22:02:38.267ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
18752025-05-08T22:02:38.268ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 9 reason = Timeout
18762025-05-08T22:02:38.268ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
18772025-05-08T22:02:38.269ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
18782025-05-08T22:02:38.269ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-05-08 21:52:38.265415839 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
18792025-05-08T22:02:38.270ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 4.568489ms iteration = 5 status = {"expiration":"2025-05-08T21:52:38.265415839Z","pruned":[]}
18802025-05-08T22:02:38.270ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18812025-05-08T22:02:38.270ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1134, tv_nsec: 12648754 } })
18822025-05-08T22:02:38.271ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163
18832025-05-08T22:02:38.271ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1134, tv_nsec: 13728525 } })
18842025-05-08T22:02:38.272ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1163
18852025-05-08T22:02:38.272ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
18862025-05-08T22:02:38.272ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
18872025-05-08T22:02:38.272ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
18882025-05-08T22:02:38.274ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 7.059625ms iteration = 9 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
18892025-05-08T22:02:38.274ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18902025-05-08T22:02:38.276ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
18912025-05-08T22:02:38.276ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
18922025-05-08T22:02:38.278ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
18932025-05-08T22:02:38.280ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18942025-05-08T22:02:38.280ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 25 reason = Timeout
18952025-05-08T22:02:38.280ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
18962025-05-08T22:02:38.280ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 9 reason = Timeout
18972025-05-08T22:02:38.280ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
18982025-05-08T22:02:38.280ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 10 reason = Timeout
18992025-05-08T22:02:38.280ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
19002025-05-08T22:02:38.280ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
19012025-05-08T22:02:38.280ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 5 reason = Timeout
19022025-05-08T22:02:38.280ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
19032025-05-08T22:02:38.280ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 9 reason = Timeout
19042025-05-08T22:02:38.280ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 5 reason = Timeout
19052025-05-08T22:02:38.280ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
19062025-05-08T22:02:38.280ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:53
19072025-05-08T22:02:38.280ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
19082025-05-08T22:02:38.280ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 9 reason = Timeout
19092025-05-08T22:02:38.280ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
19102025-05-08T22:02:38.280ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
19112025-05-08T22:02:38.280ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 9 reason = Timeout
19122025-05-08T22:02:38.280ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
19132025-05-08T22:02:38.280ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 9 reason = Timeout
19142025-05-08T22:02:38.280ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
19152025-05-08T22:02:38.280ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 9 reason = Timeout
19162025-05-08T22:02:38.280ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 9 reason = Timeout
19172025-05-08T22:02:38.280ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
19182025-05-08T22:02:38.280ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
19192025-05-08T22:02:38.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
19202025-05-08T22:02:38.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
19212025-05-08T22:02:38.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
19222025-05-08T22:02:38.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
19232025-05-08T22:02:38.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
19242025-05-08T22:02:38.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
19252025-05-08T22:02:38.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
19262025-05-08T22:02:38.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
19272025-05-08T22:02:38.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
19282025-05-08T22:02:38.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
19292025-05-08T22:02:38.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
19302025-05-08T22:02:38.282ZDEBGnexus (BackgroundTasks): activating background_task = webhook_deliverator iteration = 5 reason = Timeout
19312025-05-08T22:02:38.282ZDEBGnexus (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} }
19322025-05-08T22:02:38.282ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 9 reason = Timeout
19332025-05-08T22:02:38.282ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
19342025-05-08T22:02:38.282ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 5 reason = Timeout
19352025-05-08T22:02:38.282ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 9 reason = Timeout
19362025-05-08T22:02:38.282ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
19372025-05-08T22:02:38.282ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
19382025-05-08T22:02:38.282ZDEBGnexus (BackgroundTasks): activating background_task = webhook_dispatcher iteration = 5 reason = Timeout
19392025-05-08T22:02:38.282ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 9 reason = Timeout
19402025-05-08T22:02:38.282ZDEBGnexus (BackgroundTasks): roles background_task = webhook_dispatcher roles = RoleSet { roles: {} }
19412025-05-08T22:02:38.282ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
19422025-05-08T22:02:38.282ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 9 reason = Timeout
19432025-05-08T22:02:38.282ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
19442025-05-08T22:02:38.282ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 9 reason = Timeout
19452025-05-08T22:02:38.282ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
19462025-05-08T22:02:38.282ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 5 reason = Timeout
19472025-05-08T22:02:38.282ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
19482025-05-08T22:02:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
19492025-05-08T22:02:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
19502025-05-08T22:02:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
19512025-05-08T22:02:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
19522025-05-08T22:02:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
19532025-05-08T22:02:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database result = Ok(())
19542025-05-08T22:02:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
19552025-05-08T22:02:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database result = Ok(())
19562025-05-08T22:02:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
19572025-05-08T22:02:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
19582025-05-08T22:02:38.284ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
19592025-05-08T22:02:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
19602025-05-08T22:02:38.290ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19612025-05-08T22:02:38.294ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
19622025-05-08T22:02:38.294ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
19632025-05-08T22:02:38.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
19642025-05-08T22:02:38.297ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19652025-05-08T22:02:38.302ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
19662025-05-08T22:02:38.302ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
19672025-05-08T22:02:38.302ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
19682025-05-08T22:02:38.312ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 30.194153ms iteration = 5 status = {"errors":[],"found_port_ok":[]}
19692025-05-08T22:02:38.342ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19702025-05-08T22:02:38.345ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19712025-05-08T22:02:38.346ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
19722025-05-08T22:02:38.346ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
19732025-05-08T22:02:38.346ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
19742025-05-08T22:02:38.349ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
19752025-05-08T22:02:38.349ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
19762025-05-08T22:02:38.349ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
19772025-05-08T22:02:38.356ZDEBGnexus (BackgroundTasks): activation complete background_task = webhook_deliverator elapsed = 73.766618ms iteration = 5 status = {"by_rx":{},"error":null}
19782025-05-08T22:02:38.357ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19792025-05-08T22:02:38.357ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
19802025-05-08T22:02:38.357ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
19812025-05-08T22:02:38.357ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
19822025-05-08T22:02:38.357ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
19832025-05-08T22:02:38.358ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
19842025-05-08T22:02:38.360ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19852025-05-08T22:02:38.360ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
19862025-05-08T22:02:38.360ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
19872025-05-08T22:02:38.361ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
19882025-05-08T22:02:38.362ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19892025-05-08T22:02:38.363ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19902025-05-08T22:02:38.363ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
19912025-05-08T22:02:38.364ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
19922025-05-08T22:02:38.366ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
19932025-05-08T22:02:38.366ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
19942025-05-08T22:02:38.366ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
19952025-05-08T22:02:38.366ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
19962025-05-08T22:02:38.367ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
19972025-05-08T22:02:38.367ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
19982025-05-08T22:02:38.368ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
19992025-05-08T22:02:38.368ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20002025-05-08T22:02:38.368ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
20012025-05-08T22:02:38.368ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
20022025-05-08T22:02:38.368ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
20032025-05-08T22:02:38.368ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
20042025-05-08T22:02:38.369ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
20052025-05-08T22:02:38.369ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
20062025-05-08T22:02:38.371ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
20072025-05-08T22:02:38.371ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
20082025-05-08T22:02:38.371ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
20092025-05-08T22:02:38.373ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
20102025-05-08T22:02:38.373ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
20112025-05-08T22:02:38.374ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20122025-05-08T22:02:38.375ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 95.727184ms iteration = 25 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
20132025-05-08T22:02:38.376ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20142025-05-08T22:02:38.379ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
20152025-05-08T22:02:38.379ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
20162025-05-08T22:02:38.379ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
20172025-05-08T22:02:38.380ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
20182025-05-08T22:02:38.380ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
20192025-05-08T22:02:38.380ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
20202025-05-08T22:02:38.382ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
20212025-05-08T22:02:38.382ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
20222025-05-08T22:02:38.387ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
20232025-05-08T22:02:38.388ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
20242025-05-08T22:02:38.392ZDEBGnexus (BackgroundTasks): roles background_task = webhook_dispatcher roles = RoleSet { roles: {} }
20252025-05-08T22:02:38.392ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20262025-05-08T22:02:38.393ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database result = Ok(())
20272025-05-08T22:02:38.393ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 110.879636ms iteration = 9 status = {"errors":[],"finish_invoked_ok":[]}
20282025-05-08T22:02:38.393ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1134, tv_nsec: 135164434 } })
20292025-05-08T22:02:38.393ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
20302025-05-08T22:02:38.396ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
20312025-05-08T22:02:38.396ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
20322025-05-08T22:02:38.396ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
20332025-05-08T22:02:38.396ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
20342025-05-08T22:02:38.396ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 114.358873ms iteration = 5 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
20352025-05-08T22:02:38.397ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
20362025-05-08T22:02:38.397ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
20372025-05-08T22:02:38.397ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
20382025-05-08T22:02:38.397ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 115.379743ms iteration = 9 status = {"errors":[],"requests_created_ok":[]}
20392025-05-08T22:02:38.399ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
20402025-05-08T22:02:38.400ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
20412025-05-08T22:02:38.401ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20422025-05-08T22:02:38.404ZDEBGnexus (BackgroundTasks): activation complete background_task = webhook_dispatcher elapsed = 121.846614ms iteration = 5 status = {"dispatched":[],"errors":[],"glob_version":"140.0.0","globs_reprocessed":{},"no_receivers":[]}
20432025-05-08T22:02:38.405ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
20442025-05-08T22:02:38.405ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
20452025-05-08T22:02:38.405ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
20462025-05-08T22:02:38.420ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
20472025-05-08T22:02:38.420ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
20482025-05-08T22:02:38.420ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
20492025-05-08T22:02:38.420ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
20502025-05-08T22:02:38.420ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
20512025-05-08T22:02:38.428ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
20522025-05-08T22:02:38.428ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
20532025-05-08T22:02:38.432ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20542025-05-08T22:02:38.435ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
20552025-05-08T22:02:38.435ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
20562025-05-08T22:02:38.435ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
20572025-05-08T22:02:38.437ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 155.114999ms iteration = 9 status = {"errors":[],"garbage_collect_requested":[]}
20582025-05-08T22:02:38.438ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
20592025-05-08T22:02:38.438ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
20602025-05-08T22:02:38.438ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
20612025-05-08T22:02:38.438ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
20622025-05-08T22:02:38.440ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20632025-05-08T22:02:38.442ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
20642025-05-08T22:02:38.443ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
20652025-05-08T22:02:38.443ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
20662025-05-08T22:02:38.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
20672025-05-08T22:02:38.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
20682025-05-08T22:02:38.444ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
20692025-05-08T22:02:38.444ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 177.477948ms iteration = 25 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
20702025-05-08T22:02:38.445ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20712025-05-08T22:02:38.445ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
20722025-05-08T22:02:38.445ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
20732025-05-08T22:02:38.445ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
20742025-05-08T22:02:38.445ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:552 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
20752025-05-08T22:02:38.445ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 165.587391ms iteration = 9 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
20762025-05-08T22:02:38.445ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:362
20772025-05-08T22:02:38.445ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 176.813001ms iteration = 9 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
20782025-05-08T22:02:38.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
20792025-05-08T22:02:38.449ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 2f2f2c32-8d58-4801-a669-9618eac0c3f7, lookup_type: ById(2f2f2c32-8d58-4801-a669-9618eac0c3f7) } result = Ok(())
20802025-05-08T22:02:38.449ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
20812025-05-08T22:02:38.449ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
20822025-05-08T22:02:38.466ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
20832025-05-08T22:02:38.466ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:137 found = 0 not_ready_to_be_deleted = 0
20842025-05-08T22:02:38.466ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 186.240411ms iteration = 5 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
20852025-05-08T22:02:38.470ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 188.410105ms iteration = 9 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
20862025-05-08T22:02:38.475ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
20872025-05-08T22:02:38.476ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
20882025-05-08T22:02:38.477ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 197.127718ms iteration = 9 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
20892025-05-08T22:02:38.478ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
20902025-05-08T22:02:38.478ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 7586992a-6fc5-4073-8962-728778f16aa5 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
20912025-05-08T22:02:38.478ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
20922025-05-08T22:02:38.478ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
20932025-05-08T22:02:38.480ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20942025-05-08T22:02:38.485ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
20952025-05-08T22:02:38.485ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
20962025-05-08T22:02:38.485ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20972025-05-08T22:02:38.485ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
20982025-05-08T22:02:38.488ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 7586992a-6fc5-4073-8962-728778f16aa5 dns_zone_name = control-plane.oxide.internal found_so_far = 55 version = 1
20992025-05-08T22:02:38.488ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
21002025-05-08T22:02:38.488ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-05-08 21:59:21.182334 UTC dns_group = internal generation = 1
21012025-05-08T22:02:38.488ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 223.226317ms iteration = 5 status = {"generation":1}
21022025-05-08T22:02:38.489ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
21032025-05-08T22:02:38.489ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
21042025-05-08T22:02:38.489ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
21052025-05-08T22:02:38.493ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-05-08T21:58:35.560911Z, time_modified: 2025-05-08T21:58:35.560911Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, db_silo: Silo { identity: SiloIdentity { id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-05-08T21:59:21.709488Z, time_modified: 2025-05-08T21:59:21.709488Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, db_silo: Silo { identity: SiloIdentity { id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-05-08T21:59:21.709488Z, time_modified: 2025-05-08T21:59:21.709488Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47" }] }) }
21062025-05-08T22:02:38.493ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 228.456253ms iteration = 5 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"7137ee58-6aa7-4199-a248-feadf3b8bafb","tls_certs":["37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47"]}},"default_endpoint":{"silo_id":"7137ee58-6aa7-4199-a248-feadf3b8bafb","tls_certs":["37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
21072025-05-08T22:02:38.495ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
21082025-05-08T22:02:38.496ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
21092025-05-08T22:02:38.506ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
21102025-05-08T22:02:38.506ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
21112025-05-08T22:02:38.506ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 226.29943ms iteration = 9 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
21122025-05-08T22:02:38.529ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 246.986151ms iteration = 9 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
21132025-05-08T22:02:38.537ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
21142025-05-08T22:02:38.538ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
21152025-05-08T22:02:38.553ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
21162025-05-08T22:02:38.553ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = d8128696-db5a-4b88-914a-69b28494e50e dns_zone_name = oxide.test found_so_far = 0 version = 2
21172025-05-08T22:02:38.553ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
21182025-05-08T22:02:38.553ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
21192025-05-08T22:02:38.555ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21202025-05-08T22:02:38.558ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 278.689101ms iteration = 9 status = 0
21212025-05-08T22:02:38.559ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
21222025-05-08T22:02:38.560ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
21232025-05-08T22:02:38.560ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
21242025-05-08T22:02:38.562ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = d8128696-db5a-4b88-914a-69b28494e50e dns_zone_name = oxide.test found_so_far = 1 version = 2
21252025-05-08T22:02:38.562ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
21262025-05-08T22:02:38.562ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-05-08 21:59:21.759368 UTC dns_group = external generation = 2
21272025-05-08T22:02:38.562ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 297.040607ms iteration = 5 status = {"generation":2}
21282025-05-08T22:02:38.564ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e zone_address = fd00:1122:3344:101::2
21292025-05-08T22:02:38.564ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
21302025-05-08T22:02:38.564ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "91ab1edd-3992-4ebd-8cf3-6599714ca585", "content-length": "26", "date": "Thu, 08 May 2025 22:02:38 GMT"} })
21312025-05-08T22:02:38.564ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1174 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
21322025-05-08T22:02:38.566ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
21332025-05-08T22:02:38.598ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 318.230074ms iteration = 10 status = {"physical_disks_added":0}
21342025-05-08T22:02:38.674ZINFOnexus (BackgroundTasks): instance reincarnation completed successfully background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:113 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0
21352025-05-08T22:02:38.675ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 394.835793ms iteration = 5 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
21362025-05-08T22:02:38.756ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
21372025-05-08T22:02:38.762ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
21382025-05-08T22:02:38.762ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
21392025-05-08T22:02:38.762ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "05e13049-38ff-4be8-8a93-62b1ae7bf76f", "content-length": "26", "date": "Thu, 08 May 2025 22:02:38 GMT"} })
21402025-05-08T22:02:38.762ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
21412025-05-08T22:02:38.764ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
21422025-05-08T22:02:38.777ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
21432025-05-08T22:02:38.801ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
21442025-05-08T22:02:38.801ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
21452025-05-08T22:02:38.801ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "24976116-902d-4d0d-a6af-6d1608b97d9f", "content-length": "26", "date": "Thu, 08 May 2025 22:02:38 GMT"} })
21462025-05-08T22:02:38.801ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
21472025-05-08T22:02:38.803ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
21482025-05-08T22:02:38.814ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
21492025-05-08T22:02:38.814ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
21502025-05-08T22:02:38.816ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
21512025-05-08T22:02:38.816ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
21522025-05-08T22:02:38.817ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c83ba43-e70c-435f-a87b-607677bd4d31", "content-length": "372", "date": "Thu, 08 May 2025 22:02:38 GMT"} })
21532025-05-08T22:02:38.817ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
21542025-05-08T22:02:38.817ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
21552025-05-08T22:02:38.818ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21562025-05-08T22:02:38.820ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
21572025-05-08T22:02:38.820ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
21582025-05-08T22:02:38.821ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
21592025-05-08T22:02:38.867ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
21602025-05-08T22:02:38.867ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
21612025-05-08T22:02:38.867ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "9e191ac8-60d6-4665-9994-b81ad4774b6c", "date": "Thu, 08 May 2025 22:02:38 GMT"} })
21622025-05-08T22:02:38.867ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
21632025-05-08T22:02:38.867ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21642025-05-08T22:02:38.867ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
21652025-05-08T22:02:38.869ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
21662025-05-08T22:02:38.869ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
21672025-05-08T22:02:38.869ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
21682025-05-08T22:02:38.870ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
21692025-05-08T22:02:38.870ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
21702025-05-08T22:02:38.874ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
21712025-05-08T22:02:38.874ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
21722025-05-08T22:02:38.877ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
21732025-05-08T22:02:38.877ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
21742025-05-08T22:02:38.877ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "cff46fb7-c235-465b-9522-cc911e80de50", "date": "Thu, 08 May 2025 22:02:38 GMT"} })
21752025-05-08T22:02:38.880ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 600.116839ms iteration = 9 status = {}
21762025-05-08T22:02:38.958ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 693.525657ms iteration = 9 status = {}
21772025-05-08T22:02:39.048ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
21782025-05-08T22:02:39.048ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
21792025-05-08T22:02:39.096ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
21802025-05-08T22:02:39.099ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
21812025-05-08T22:02:39.100ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "6eeb0eec-f8e3-42c2-ade5-04afabc136c5", "date": "Thu, 08 May 2025 22:02:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
21822025-05-08T22:02:39.100ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
21832025-05-08T22:02:39.102ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "e64ac698-ef36-48c9-a7fa-6ba9b69fa852", "date": "Thu, 08 May 2025 22:02:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
21842025-05-08T22:02:39.102ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
21852025-05-08T22:02:39.104ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
21862025-05-08T22:02:39.104ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 839.035524ms iteration = 7 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
21872025-05-08T22:02:39.167ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
21882025-05-08T22:02:39.168ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
21892025-05-08T22:02:39.181ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
21902025-05-08T22:02:39.182ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "434f7039-3d40-4e5f-8cde-3cec4ba24d5f", "content-length": "691", "date": "Thu, 08 May 2025 22:02:39 GMT"} })
21912025-05-08T22:02:39.182ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
21922025-05-08T22:02:39.182ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
21932025-05-08T22:02:39.182ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
21942025-05-08T22:02:39.184ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 918.617565ms iteration = 9 status = null
21952025-05-08T22:02:39.184ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 904.669977ms iteration = 9 status = {}
21962025-05-08T22:02:39.185ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: cd0b930c-534b-46ad-ac3b-2eaa94b0bf2c, rack_id: 64f3d6f4-89b8-48aa-a622-b684e261667e, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(65f83a36-18fb-49b7-9914-87c8517d200e) } rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
21972025-05-08T22:02:39.185ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
21982025-05-08T22:02:39.185ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
21992025-05-08T22:02:39.197ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1833 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
22002025-05-08T22:02:39.197ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1860 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 65f83a36-18fb-49b7-9914-87c8517d200e
22012025-05-08T22:02:39.197ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
22022025-05-08T22:02:39.197ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
22032025-05-08T22:02:39.197ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
22042025-05-08T22:02:39.197ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
22052025-05-08T22:02:39.197ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
22062025-05-08T22:02:39.197ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2007 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
22072025-05-08T22:02:39.197ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
22082025-05-08T22:02:39.197ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2043 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
22092025-05-08T22:02:39.197ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
22102025-05-08T22:02:39.197ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
22112025-05-08T22:02:39.198ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
22122025-05-08T22:02:39.199ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
22132025-05-08T22:02:39.199ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "0e4205c9-000b-43e0-9b44-c009ca3ec414", "date": "Thu, 08 May 2025 22:02:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
22142025-05-08T22:02:39.199ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
22152025-05-08T22:02:39.201ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "804eb8df-53e9-4eb3-8eaf-21a34b7bbd9a", "date": "Thu, 08 May 2025 22:02:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
22162025-05-08T22:02:39.201ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
22172025-05-08T22:02:39.202ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
22182025-05-08T22:02:39.202ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
22192025-05-08T22:02:39.203ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "c2d7020a-c9fd-4d4e-a6cd-d95b0aa28133", "date": "Thu, 08 May 2025 22:02:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
22202025-05-08T22:02:39.203ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
22212025-05-08T22:02:39.204ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
22222025-05-08T22:02:39.204ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 939.295376ms iteration = 7 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
22232025-05-08T22:02:39.300ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "23417ae3-c0db-4e80-86bf-de9c2540f433", "date": "Thu, 08 May 2025 22:02:39 GMT"} })
22242025-05-08T22:02:39.300ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch = Switch0
22252025-05-08T22:02:39.300ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
22262025-05-08T22:02:39.300ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "d08dcf04-4be4-41c1-b2d1-5163d3b2a831", "content-length": "682", "date": "Thu, 08 May 2025 22:02:38 GMT"} })
22272025-05-08T22:02:39.300ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
22282025-05-08T22:02:39.300ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
22292025-05-08T22:02:39.304ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
22302025-05-08T22:02:39.304ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
22312025-05-08T22:02:39.308ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
22322025-05-08T22:02:39.308ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
22332025-05-08T22:02:39.310ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
22342025-05-08T22:02:39.311ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
22352025-05-08T22:02:39.318ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 1.038772141s iteration = 9 status = {}
22362025-05-08T22:02:41.935ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 282 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 21b34b22-ed4b-47cf-b14a-028cc20d6e42 response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
22372025-05-08T22:02:48.267ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 26 reason = Timeout
22382025-05-08T22:02:48.267ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
22392025-05-08T22:02:48.267ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
22402025-05-08T22:02:48.269ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22412025-05-08T22:02:48.272ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
22422025-05-08T22:02:48.272ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
22432025-05-08T22:02:48.272ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
22442025-05-08T22:02:48.275ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
22452025-05-08T22:02:48.275ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
22462025-05-08T22:02:48.277ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22472025-05-08T22:02:48.279ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
22482025-05-08T22:02:48.279ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
22492025-05-08T22:02:48.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
22502025-05-08T22:02:48.280ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 26 reason = Timeout
22512025-05-08T22:02:48.280ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
22522025-05-08T22:02:48.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
22532025-05-08T22:02:48.290ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
22542025-05-08T22:02:48.290ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.489201ms iteration = 26 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
22552025-05-08T22:02:48.312ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
22562025-05-08T22:02:48.312ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
22572025-05-08T22:02:48.348ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 68.389601ms iteration = 26 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
22582025-05-08T22:02:51.935ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 272 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 740422e2-6728-4301-9896-fe1146924723 response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
22592025-05-08T22:02:58.267ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 27 reason = Timeout
22602025-05-08T22:02:58.267ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
22612025-05-08T22:02:58.267ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
22622025-05-08T22:02:58.269ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22632025-05-08T22:02:58.272ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
22642025-05-08T22:02:58.272ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
22652025-05-08T22:02:58.272ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
22662025-05-08T22:02:58.273ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
22672025-05-08T22:02:58.274ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
22682025-05-08T22:02:58.275ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22692025-05-08T22:02:58.278ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
22702025-05-08T22:02:58.278ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
22712025-05-08T22:02:58.278ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
22722025-05-08T22:02:58.280ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 27 reason = Timeout
22732025-05-08T22:02:58.280ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
22742025-05-08T22:02:58.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
22752025-05-08T22:02:58.282ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
22762025-05-08T22:02:58.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
22772025-05-08T22:02:58.314ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 33.959793ms iteration = 27 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
22782025-05-08T22:02:58.381ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
22792025-05-08T22:02:58.381ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 114.450124ms iteration = 27 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
22802025-05-08T22:03:01.935ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 292 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 5a9d56a2-b601-4c1a-9426-5368d3e0cac8 response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
22812025-05-08T22:03:08.265ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 10 reason = Timeout
22822025-05-08T22:03:08.265ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
22832025-05-08T22:03:08.265ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 10 reason = Timeout
22842025-05-08T22:03:08.265ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
22852025-05-08T22:03:08.266ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
22862025-05-08T22:03:08.266ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
22872025-05-08T22:03:08.267ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 28 reason = Timeout
22882025-05-08T22:03:08.267ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 10 reason = Timeout
22892025-05-08T22:03:08.267ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
22902025-05-08T22:03:08.268ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
22912025-05-08T22:03:08.269ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1164, tv_nsec: 10720233 } })
22922025-05-08T22:03:08.269ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 10 reason = Timeout
22932025-05-08T22:03:08.269ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
22942025-05-08T22:03:08.269ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1164, tv_nsec: 10896635 } })
22952025-05-08T22:03:08.269ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1163
22962025-05-08T22:03:08.269ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
22972025-05-08T22:03:08.269ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163
22982025-05-08T22:03:08.270ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22992025-05-08T22:03:08.271ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23002025-05-08T22:03:08.271ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.923952ms iteration = 10 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
23012025-05-08T22:03:08.273ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
23022025-05-08T22:03:08.273ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
23032025-05-08T22:03:08.273ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
23042025-05-08T22:03:08.273ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
23052025-05-08T22:03:08.273ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
23062025-05-08T22:03:08.273ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
23072025-05-08T22:03:08.280ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 11 reason = Timeout
23082025-05-08T22:03:08.280ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 10 reason = Timeout
23092025-05-08T22:03:08.280ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
23102025-05-08T22:03:08.280ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
23112025-05-08T22:03:08.280ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 10 reason = Timeout
23122025-05-08T22:03:08.280ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
23132025-05-08T22:03:08.280ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
23142025-05-08T22:03:08.280ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 10 reason = Timeout
23152025-05-08T22:03:08.280ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
23162025-05-08T22:03:08.280ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 10 reason = Timeout
23172025-05-08T22:03:08.280ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
23182025-05-08T22:03:08.280ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 10 reason = Timeout
23192025-05-08T22:03:08.280ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
23202025-05-08T22:03:08.280ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 10 reason = Timeout
23212025-05-08T22:03:08.280ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
23222025-05-08T22:03:08.280ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 28 reason = Timeout
23232025-05-08T22:03:08.280ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
23242025-05-08T22:03:08.280ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 10 reason = Timeout
23252025-05-08T22:03:08.280ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
23262025-05-08T22:03:08.280ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
23272025-05-08T22:03:08.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
23282025-05-08T22:03:08.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
23292025-05-08T22:03:08.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
23302025-05-08T22:03:08.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
23312025-05-08T22:03:08.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
23322025-05-08T22:03:08.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
23332025-05-08T22:03:08.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
23342025-05-08T22:03:08.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
23352025-05-08T22:03:08.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
23362025-05-08T22:03:08.282ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 10 reason = Timeout
23372025-05-08T22:03:08.282ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
23382025-05-08T22:03:08.282ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 10 reason = Timeout
23392025-05-08T22:03:08.282ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
23402025-05-08T22:03:08.282ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 10 reason = Timeout
23412025-05-08T22:03:08.282ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
23422025-05-08T22:03:08.282ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 10 reason = Timeout
23432025-05-08T22:03:08.282ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
23442025-05-08T22:03:08.282ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 10 reason = Timeout
23452025-05-08T22:03:08.282ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
23462025-05-08T22:03:08.282ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23472025-05-08T22:03:08.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
23482025-05-08T22:03:08.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
23492025-05-08T22:03:08.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
23502025-05-08T22:03:08.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
23512025-05-08T22:03:08.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
23522025-05-08T22:03:08.284ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
23532025-05-08T22:03:08.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
23542025-05-08T22:03:08.284ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23552025-05-08T22:03:08.284ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
23562025-05-08T22:03:08.284ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
23572025-05-08T22:03:08.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
23582025-05-08T22:03:08.287ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23592025-05-08T22:03:08.288ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
23602025-05-08T22:03:08.288ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
23612025-05-08T22:03:08.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
23622025-05-08T22:03:08.289ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
23632025-05-08T22:03:08.289ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
23642025-05-08T22:03:08.290ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
23652025-05-08T22:03:08.299ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 17.106593ms iteration = 10 status = {"errors":[],"requests_created_ok":[]}
23662025-05-08T22:03:08.299ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1164, tv_nsec: 41430942 } })
23672025-05-08T22:03:08.300ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
23682025-05-08T22:03:08.300ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:552 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
23692025-05-08T22:03:08.300ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 20.200736ms iteration = 10 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
23702025-05-08T22:03:08.337ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
23712025-05-08T22:03:08.337ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
23722025-05-08T22:03:08.341ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
23732025-05-08T22:03:08.341ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
23742025-05-08T22:03:08.341ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
23752025-05-08T22:03:08.341ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
23762025-05-08T22:03:08.343ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
23772025-05-08T22:03:08.343ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
23782025-05-08T22:03:08.346ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 65.814794ms iteration = 28 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
23792025-05-08T22:03:08.346ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23802025-05-08T22:03:08.349ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
23812025-05-08T22:03:08.349ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
23822025-05-08T22:03:08.350ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
23832025-05-08T22:03:08.350ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 68.038538ms iteration = 10 status = {"errors":[],"garbage_collect_requested":[]}
23842025-05-08T22:03:08.352ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23852025-05-08T22:03:08.352ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
23862025-05-08T22:03:08.352ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
23872025-05-08T22:03:08.352ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
23882025-05-08T22:03:08.352ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
23892025-05-08T22:03:08.352ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "9788de4c-ad83-46e5-946f-889a4ac71309", "content-length": "26", "date": "Thu, 08 May 2025 22:03:08 GMT"} })
23902025-05-08T22:03:08.352ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
23912025-05-08T22:03:08.353ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
23922025-05-08T22:03:08.353ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
23932025-05-08T22:03:08.354ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
23942025-05-08T22:03:08.356ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
23952025-05-08T22:03:08.356ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
23962025-05-08T22:03:08.356ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
23972025-05-08T22:03:08.358ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
23982025-05-08T22:03:08.359ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
23992025-05-08T22:03:08.359ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24002025-05-08T22:03:08.360ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 78.114885ms iteration = 10 status = {"errors":[],"finish_invoked_ok":[]}
24012025-05-08T22:03:08.361ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
24022025-05-08T22:03:08.361ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
24032025-05-08T22:03:08.361ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 81.461581ms iteration = 10 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
24042025-05-08T22:03:08.362ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
24052025-05-08T22:03:08.362ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
24062025-05-08T22:03:08.362ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
24072025-05-08T22:03:08.382ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24082025-05-08T22:03:08.382ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24092025-05-08T22:03:08.384ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24102025-05-08T22:03:08.385ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
24112025-05-08T22:03:08.385ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
24122025-05-08T22:03:08.385ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
24132025-05-08T22:03:08.385ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
24142025-05-08T22:03:08.386ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24152025-05-08T22:03:08.387ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
24162025-05-08T22:03:08.387ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24172025-05-08T22:03:08.387ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24182025-05-08T22:03:08.389ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
24192025-05-08T22:03:08.389ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
24202025-05-08T22:03:08.389ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
24212025-05-08T22:03:08.395ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
24222025-05-08T22:03:08.396ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
24232025-05-08T22:03:08.400ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24242025-05-08T22:03:08.400ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 118.298365ms iteration = 10 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
24252025-05-08T22:03:08.402ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
24262025-05-08T22:03:08.402ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
24272025-05-08T22:03:08.402ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
24282025-05-08T22:03:08.424ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:362
24292025-05-08T22:03:08.424ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 155.573184ms iteration = 10 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
24302025-05-08T22:03:08.425ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
24312025-05-08T22:03:08.425ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
24322025-05-08T22:03:08.425ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 144.858379ms iteration = 10 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
24332025-05-08T22:03:08.430ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
24342025-05-08T22:03:08.431ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
24352025-05-08T22:03:08.435ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 153.188408ms iteration = 10 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
24362025-05-08T22:03:08.443ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
24372025-05-08T22:03:08.444ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
24382025-05-08T22:03:08.444ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e6534dc-5588-4ca8-909a-dc4a96efe4a3", "content-length": "26", "date": "Thu, 08 May 2025 22:03:08 GMT"} })
24392025-05-08T22:03:08.444ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
24402025-05-08T22:03:08.445ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
24412025-05-08T22:03:08.457ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 191.74476ms iteration = 10 status = {}
24422025-05-08T22:03:08.458ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e zone_address = fd00:1122:3344:101::2
24432025-05-08T22:03:08.458ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
24442025-05-08T22:03:08.458ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "5144460f-670d-4bf4-8ff9-c533e83bc2f5", "content-length": "26", "date": "Thu, 08 May 2025 22:03:08 GMT"} })
24452025-05-08T22:03:08.458ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1174 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
24462025-05-08T22:03:08.461ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
24472025-05-08T22:03:08.502ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
24482025-05-08T22:03:08.502ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 234.987933ms iteration = 28 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
24492025-05-08T22:03:08.513ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 233.001182ms iteration = 11 status = {"physical_disks_added":0}
24502025-05-08T22:03:08.525ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
24512025-05-08T22:03:08.525ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
24522025-05-08T22:03:08.526ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "524d458c-202c-4a7c-9080-d24236e4a194", "content-length": "372", "date": "Thu, 08 May 2025 22:03:08 GMT"} })
24532025-05-08T22:03:08.526ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
24542025-05-08T22:03:08.526ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
24552025-05-08T22:03:08.546ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
24562025-05-08T22:03:08.546ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
24572025-05-08T22:03:08.549ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
24582025-05-08T22:03:08.549ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
24592025-05-08T22:03:08.549ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "6452c1e7-e733-4139-851b-c3d920e6549a", "date": "Thu, 08 May 2025 22:03:08 GMT"} })
24602025-05-08T22:03:08.549ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
24612025-05-08T22:03:08.549ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 283.808156ms iteration = 10 status = null
24622025-05-08T22:03:08.549ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
24632025-05-08T22:03:08.554ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
24642025-05-08T22:03:08.555ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
24652025-05-08T22:03:08.561ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
24662025-05-08T22:03:08.561ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
24672025-05-08T22:03:08.564ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
24682025-05-08T22:03:08.564ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
24692025-05-08T22:03:08.565ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "abe8268d-9bee-46dc-b471-a56455e0f64f", "date": "Thu, 08 May 2025 22:03:08 GMT"} })
24702025-05-08T22:03:08.566ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 286.390303ms iteration = 10 status = {}
24712025-05-08T22:03:08.590ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
24722025-05-08T22:03:08.590ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
24732025-05-08T22:03:08.592ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
24742025-05-08T22:03:08.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
24752025-05-08T22:03:08.594ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24762025-05-08T22:03:08.597ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 2f2f2c32-8d58-4801-a669-9618eac0c3f7, lookup_type: ById(2f2f2c32-8d58-4801-a669-9618eac0c3f7) } result = Ok(())
24772025-05-08T22:03:08.597ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
24782025-05-08T22:03:08.597ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
24792025-05-08T22:03:08.604ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 324.536762ms iteration = 10 status = 0
24802025-05-08T22:03:08.617ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24812025-05-08T22:03:08.618ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24822025-05-08T22:03:08.619ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: cd0b930c-534b-46ad-ac3b-2eaa94b0bf2c, rack_id: 64f3d6f4-89b8-48aa-a622-b684e261667e, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(65f83a36-18fb-49b7-9914-87c8517d200e) } rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
24832025-05-08T22:03:08.620ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24842025-05-08T22:03:08.620ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24852025-05-08T22:03:08.628ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1833 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
24862025-05-08T22:03:08.628ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1860 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 65f83a36-18fb-49b7-9914-87c8517d200e
24872025-05-08T22:03:08.628ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
24882025-05-08T22:03:08.628ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
24892025-05-08T22:03:08.628ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
24902025-05-08T22:03:08.628ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
24912025-05-08T22:03:08.628ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
24922025-05-08T22:03:08.628ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2007 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
24932025-05-08T22:03:08.629ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
24942025-05-08T22:03:08.629ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2043 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
24952025-05-08T22:03:08.629ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
24962025-05-08T22:03:08.629ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24972025-05-08T22:03:08.629ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24982025-05-08T22:03:08.631ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
24992025-05-08T22:03:08.631ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
25002025-05-08T22:03:08.675ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
25012025-05-08T22:03:08.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
25022025-05-08T22:03:08.676ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25032025-05-08T22:03:08.679ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
25042025-05-08T22:03:08.679ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
25052025-05-08T22:03:08.679ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
25062025-05-08T22:03:08.680ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25072025-05-08T22:03:08.682ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
25082025-05-08T22:03:08.682ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
25092025-05-08T22:03:08.682ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
25102025-05-08T22:03:08.731ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "5a6d2d1b-20fd-4b7a-a5de-ba45b14c83ab", "date": "Thu, 08 May 2025 22:03:08 GMT"} })
25112025-05-08T22:03:08.731ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch = Switch0
25122025-05-08T22:03:08.731ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
25132025-05-08T22:03:08.731ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "d952ca8c-599d-42da-b4ba-5940d3312f86", "content-length": "682", "date": "Thu, 08 May 2025 22:03:08 GMT"} })
25142025-05-08T22:03:08.732ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
25152025-05-08T22:03:08.732ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
25162025-05-08T22:03:08.735ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
25172025-05-08T22:03:08.735ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
25182025-05-08T22:03:08.777ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
25192025-05-08T22:03:08.777ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "497a0f00-d1c0-447a-b434-6fdd10baf9c4", "content-length": "691", "date": "Thu, 08 May 2025 22:03:08 GMT"} })
25202025-05-08T22:03:08.777ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
25212025-05-08T22:03:08.778ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
25222025-05-08T22:03:08.778ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
25232025-05-08T22:03:08.780ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 499.708825ms iteration = 10 status = {}
25242025-05-08T22:03:08.839ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
25252025-05-08T22:03:08.839ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
25262025-05-08T22:03:08.841ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
25272025-05-08T22:03:08.841ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
25282025-05-08T22:03:08.848ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 568.39843ms iteration = 10 status = {}
25292025-05-08T22:03:11.935ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 311 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = f816aea7-400d-41ea-a204-87b4a259f56d response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
25302025-05-08T22:03:18.267ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 29 reason = Timeout
25312025-05-08T22:03:18.267ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25322025-05-08T22:03:18.268ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25332025-05-08T22:03:18.270ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25342025-05-08T22:03:18.273ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
25352025-05-08T22:03:18.273ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25362025-05-08T22:03:18.273ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25372025-05-08T22:03:18.274ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25382025-05-08T22:03:18.275ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25392025-05-08T22:03:18.276ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25402025-05-08T22:03:18.279ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
25412025-05-08T22:03:18.279ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25422025-05-08T22:03:18.279ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25432025-05-08T22:03:18.280ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 29 reason = Timeout
25442025-05-08T22:03:18.280ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
25452025-05-08T22:03:18.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
25462025-05-08T22:03:18.287ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
25472025-05-08T22:03:18.287ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.127815ms iteration = 29 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
25482025-05-08T22:03:18.310ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
25492025-05-08T22:03:18.310ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
25502025-05-08T22:03:18.348ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 67.670474ms iteration = 29 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
25512025-05-08T22:03:21.935ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 277 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = f34918f6-dd1a-4c99-9cd8-238ad116959c response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
25522025-05-08T22:03:28.267ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 30 reason = Timeout
25532025-05-08T22:03:28.267ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25542025-05-08T22:03:28.267ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25552025-05-08T22:03:28.269ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25562025-05-08T22:03:28.272ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
25572025-05-08T22:03:28.272ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25582025-05-08T22:03:28.272ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25592025-05-08T22:03:28.275ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25602025-05-08T22:03:28.275ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25612025-05-08T22:03:28.277ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25622025-05-08T22:03:28.280ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
25632025-05-08T22:03:28.280ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25642025-05-08T22:03:28.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25652025-05-08T22:03:28.280ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 30 reason = Timeout
25662025-05-08T22:03:28.280ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
25672025-05-08T22:03:28.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
25682025-05-08T22:03:28.282ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
25692025-05-08T22:03:28.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
25702025-05-08T22:03:28.283ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.196884ms iteration = 30 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
25712025-05-08T22:03:28.289ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
25722025-05-08T22:03:28.289ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.340259ms iteration = 30 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
25732025-05-08T22:03:31.936ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 440 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = a3c03501-81b5-4786-b917-b621bedb000d response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
25742025-05-08T22:03:38.266ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 8 reason = Timeout
25752025-05-08T22:03:38.266ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 11 reason = Timeout
25762025-05-08T22:03:38.266ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
25772025-05-08T22:03:38.266ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 6 reason = Timeout
25782025-05-08T22:03:38.266ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
25792025-05-08T22:03:38.266ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
25802025-05-08T22:03:38.266ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 6 reason = Timeout
25812025-05-08T22:03:38.266ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 6 reason = Timeout
25822025-05-08T22:03:38.266ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 6 reason = Timeout
25832025-05-08T22:03:38.266ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
25842025-05-08T22:03:38.266ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-05-08 21:53:38.266435685 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
25852025-05-08T22:03:38.266ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
25862025-05-08T22:03:38.266ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 6 reason = Timeout
25872025-05-08T22:03:38.266ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
25882025-05-08T22:03:38.266ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
25892025-05-08T22:03:38.266ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 6 reason = Timeout
25902025-05-08T22:03:38.266ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 8 reason = Timeout
25912025-05-08T22:03:38.266ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 11 reason = Timeout
25922025-05-08T22:03:38.266ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
25932025-05-08T22:03:38.266ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
25942025-05-08T22:03:38.266ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
25952025-05-08T22:03:38.266ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
25962025-05-08T22:03:38.266ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("0ccabf41-81cd-45e7-bb26-335edaa2ee47.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("38a08dd6-6a75-4558-b4f8-df853f4d03d9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("38a08dd6-6a75-4558-b4f8-df853f4d03d9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::9))) }], valid_until: Instant { tv_sec: 1194, tv_nsec: 8252167 } })
25972025-05-08T22:03:38.266ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("228ed1c1-3cc1-4055-aae9-5595ad856b86.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("8b245480-f91d-406b-bb08-850e1c3e3fc5.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c8f20f67-ea63-4dbc-b0f1-c30876daff82.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c8f20f67-ea63-4dbc-b0f1-c30876daff82.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 1194, tv_nsec: 8256037 } })
25982025-05-08T22:03:38.267ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
25992025-05-08T22:03:38.267ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
26002025-05-08T22:03:38.267ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
26012025-05-08T22:03:38.267ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 31 reason = Timeout
26022025-05-08T22:03:38.267ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 7 reason = Timeout
26032025-05-08T22:03:38.267ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:85 reason = blueprint disabled target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2
26042025-05-08T22:03:38.267ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
26052025-05-08T22:03:38.267ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 8.37\u{b5}s iteration = 7 status = {"enabled":false,"target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2"}
26062025-05-08T22:03:38.267ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 11 reason = Timeout
26072025-05-08T22:03:38.267ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } dns_group = external
26082025-05-08T22:03:38.267ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.032871ms iteration = 6 status = {"addresses":["[fd00:1122:3344:101::8]:5353","[fd00:1122:3344:101::9]:5353"]}
26092025-05-08T22:03:38.267ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
26102025-05-08T22:03:38.267ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.128981ms iteration = 6 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:1::1]:5353"]}
26112025-05-08T22:03:38.267ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
26122025-05-08T22:03:38.268ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
26132025-05-08T22:03:38.268ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
26142025-05-08T22:03:38.268ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1194, tv_nsec: 10273829 } })
26152025-05-08T22:03:38.269ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1163
26162025-05-08T22:03:38.269ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 11 reason = Timeout
26172025-05-08T22:03:38.269ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26182025-05-08T22:03:38.269ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
26192025-05-08T22:03:38.269ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
26202025-05-08T22:03:38.270ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
26212025-05-08T22:03:38.270ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
26222025-05-08T22:03:38.270ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-05-08 21:53:38.266435685 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
26232025-05-08T22:03:38.270ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 4.244295ms iteration = 6 status = {"expiration":"2025-05-08T21:53:38.266435685Z","pruned":[]}
26242025-05-08T22:03:38.271ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1194, tv_nsec: 12833746 } })
26252025-05-08T22:03:38.271ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163
26262025-05-08T22:03:38.271ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
26272025-05-08T22:03:38.271ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
26282025-05-08T22:03:38.272ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
26292025-05-08T22:03:38.274ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26302025-05-08T22:03:38.274ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26312025-05-08T22:03:38.274ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26322025-05-08T22:03:38.275ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 8.161507ms iteration = 11 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
26332025-05-08T22:03:38.275ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26342025-05-08T22:03:38.276ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
26352025-05-08T22:03:38.277ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
26362025-05-08T22:03:38.278ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
26372025-05-08T22:03:38.278ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
26382025-05-08T22:03:38.278ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
26392025-05-08T22:03:38.279ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
26402025-05-08T22:03:38.279ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
26412025-05-08T22:03:38.279ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
26422025-05-08T22:03:38.279ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
26432025-05-08T22:03:38.280ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
26442025-05-08T22:03:38.280ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26452025-05-08T22:03:38.280ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 31 reason = Timeout
26462025-05-08T22:03:38.280ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
26472025-05-08T22:03:38.280ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 11 reason = Timeout
26482025-05-08T22:03:38.280ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
26492025-05-08T22:03:38.280ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 6 reason = Timeout
26502025-05-08T22:03:38.280ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:53
26512025-05-08T22:03:38.280ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 12 reason = Timeout
26522025-05-08T22:03:38.280ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
26532025-05-08T22:03:38.280ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 11 reason = Timeout
26542025-05-08T22:03:38.280ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
26552025-05-08T22:03:38.280ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
26562025-05-08T22:03:38.280ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 6 reason = Timeout
26572025-05-08T22:03:38.280ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
26582025-05-08T22:03:38.280ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 11 reason = Timeout
26592025-05-08T22:03:38.280ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
26602025-05-08T22:03:38.280ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
26612025-05-08T22:03:38.280ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 11 reason = Timeout
26622025-05-08T22:03:38.280ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 11 reason = Timeout
26632025-05-08T22:03:38.280ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
26642025-05-08T22:03:38.280ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
26652025-05-08T22:03:38.280ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 3 reason = Timeout
26662025-05-08T22:03:38.281ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
26672025-05-08T22:03:38.281ZDEBGnexus (BackgroundTasks): activating background_task = service_firewall_rule_propagation iteration = 2 reason = Timeout
26682025-05-08T22:03:38.281ZDEBGnexus (service-firewall-rule-progator): starting background task for service firewall rule propagation background_task = service_firewall_rule_propagation
26692025-05-08T22:03:38.281ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
26702025-05-08T22:03:38.281ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 11 reason = Timeout
26712025-05-08T22:03:38.281ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
26722025-05-08T22:03:38.281ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 11 reason = Timeout
26732025-05-08T22:03:38.281ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
26742025-05-08T22:03:38.281ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
26752025-05-08T22:03:38.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
26762025-05-08T22:03:38.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
26772025-05-08T22:03:38.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
26782025-05-08T22:03:38.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
26792025-05-08T22:03:38.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
26802025-05-08T22:03:38.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
26812025-05-08T22:03:38.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
26822025-05-08T22:03:38.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
26832025-05-08T22:03:38.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
26842025-05-08T22:03:38.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
26852025-05-08T22:03:38.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
26862025-05-08T22:03:38.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
26872025-05-08T22:03:38.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
26882025-05-08T22:03:38.281ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
26892025-05-08T22:03:38.281ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
26902025-05-08T22:03:38.281ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
26912025-05-08T22:03:38.281ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
26922025-05-08T22:03:38.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
26932025-05-08T22:03:38.281ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
26942025-05-08T22:03:38.282ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
26952025-05-08T22:03:38.282ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
26962025-05-08T22:03:38.282ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
26972025-05-08T22:03:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
26982025-05-08T22:03:38.283ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 11 reason = Timeout
26992025-05-08T22:03:38.283ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
27002025-05-08T22:03:38.283ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 11 reason = Timeout
27012025-05-08T22:03:38.283ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
27022025-05-08T22:03:38.283ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 6 reason = Timeout
27032025-05-08T22:03:38.283ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
27042025-05-08T22:03:38.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
27052025-05-08T22:03:38.283ZDEBGnexus (BackgroundTasks): activating background_task = webhook_deliverator iteration = 6 reason = Timeout
27062025-05-08T22:03:38.283ZDEBGnexus (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} }
27072025-05-08T22:03:38.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
27082025-05-08T22:03:38.283ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 6 reason = Timeout
27092025-05-08T22:03:38.283ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
27102025-05-08T22:03:38.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
27112025-05-08T22:03:38.283ZDEBGnexus (BackgroundTasks): activating background_task = webhook_dispatcher iteration = 6 reason = Timeout
27122025-05-08T22:03:38.284ZDEBGnexus (BackgroundTasks): roles background_task = webhook_dispatcher roles = RoleSet { roles: {} }
27132025-05-08T22:03:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
27142025-05-08T22:03:38.284ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 11 reason = Timeout
27152025-05-08T22:03:38.284ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
27162025-05-08T22:03:38.284ZDEBGnexus (BackgroundTasks): activating background_task = tuf_artifact_replication iteration = 2 reason = Timeout
27172025-05-08T22:03:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database result = Ok(())
27182025-05-08T22:03:38.284ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 11 reason = Timeout
27192025-05-08T22:03:38.284ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
27202025-05-08T22:03:38.284ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
27212025-05-08T22:03:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database result = Ok(())
27222025-05-08T22:03:38.284ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 11 reason = Timeout
27232025-05-08T22:03:38.284ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
27242025-05-08T22:03:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
27252025-05-08T22:03:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
27262025-05-08T22:03:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
27272025-05-08T22:03:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
27282025-05-08T22:03:38.285ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
27292025-05-08T22:03:38.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
27302025-05-08T22:03:38.285ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
27312025-05-08T22:03:38.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
27322025-05-08T22:03:38.286ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27332025-05-08T22:03:38.287ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
27342025-05-08T22:03:38.287ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
27352025-05-08T22:03:38.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
27362025-05-08T22:03:38.288ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
27372025-05-08T22:03:38.289ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
27382025-05-08T22:03:38.289ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
27392025-05-08T22:03:38.289ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27402025-05-08T22:03:38.290ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27412025-05-08T22:03:38.291ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27422025-05-08T22:03:38.292ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
27432025-05-08T22:03:38.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
27442025-05-08T22:03:38.293ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
27452025-05-08T22:03:38.293ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
27462025-05-08T22:03:38.294ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
27472025-05-08T22:03:38.294ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
27482025-05-08T22:03:38.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
27492025-05-08T22:03:38.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
27502025-05-08T22:03:38.295ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
27512025-05-08T22:03:38.295ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
27522025-05-08T22:03:38.295ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
27532025-05-08T22:03:38.296ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27542025-05-08T22:03:38.297ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
27552025-05-08T22:03:38.297ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
27562025-05-08T22:03:38.298ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27572025-05-08T22:03:38.301ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Inventory result = Ok(())
27582025-05-08T22:03:38.301ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
27592025-05-08T22:03:38.301ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
27602025-05-08T22:03:38.303ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
27612025-05-08T22:03:38.303ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
27622025-05-08T22:03:38.303ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27632025-05-08T22:03:38.303ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
27642025-05-08T22:03:38.304ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27652025-05-08T22:03:38.304ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27662025-05-08T22:03:38.307ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
27672025-05-08T22:03:38.307ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
27682025-05-08T22:03:38.307ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
27692025-05-08T22:03:38.309ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
27702025-05-08T22:03:38.309ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
27712025-05-08T22:03:38.310ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
27722025-05-08T22:03:38.310ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 27.13687ms iteration = 11 status = {"errors":[],"garbage_collect_requested":[]}
27732025-05-08T22:03:38.310ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
27742025-05-08T22:03:38.310ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
27752025-05-08T22:03:38.311ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
27762025-05-08T22:03:38.316ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
27772025-05-08T22:03:38.316ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
27782025-05-08T22:03:38.316ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 33.439468ms iteration = 6 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
27792025-05-08T22:03:38.328ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 44.548206ms iteration = 6 status = {"errors":[],"found_port_ok":[]}
27802025-05-08T22:03:38.329ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 45.73202ms iteration = 11 status = {"errors":[],"finish_invoked_ok":[]}
27812025-05-08T22:03:38.331ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27822025-05-08T22:03:38.334ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 53.2206ms iteration = 11 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
27832025-05-08T22:03:38.335ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
27842025-05-08T22:03:38.336ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
27852025-05-08T22:03:38.336ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
27862025-05-08T22:03:38.336ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
27872025-05-08T22:03:38.336ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
27882025-05-08T22:03:38.336ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
27892025-05-08T22:03:38.338ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
27902025-05-08T22:03:38.338ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
27912025-05-08T22:03:38.338ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
27922025-05-08T22:03:38.341ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
27932025-05-08T22:03:38.341ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27942025-05-08T22:03:38.341ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
27952025-05-08T22:03:38.342ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
27962025-05-08T22:03:38.342ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
27972025-05-08T22:03:38.343ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1194, tv_nsec: 84604024 } })
27982025-05-08T22:03:38.343ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
27992025-05-08T22:03:38.343ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
28002025-05-08T22:03:38.344ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 7586992a-6fc5-4073-8962-728778f16aa5 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
28012025-05-08T22:03:38.344ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
28022025-05-08T22:03:38.344ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
28032025-05-08T22:03:38.344ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
28042025-05-08T22:03:38.344ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
28052025-05-08T22:03:38.344ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
28062025-05-08T22:03:38.348ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28072025-05-08T22:03:38.350ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28082025-05-08T22:03:38.352ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
28092025-05-08T22:03:38.352ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
28102025-05-08T22:03:38.352ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
28112025-05-08T22:03:38.354ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
28122025-05-08T22:03:38.354ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
28132025-05-08T22:03:38.354ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
28142025-05-08T22:03:38.362ZDEBGnexus (BackgroundTasks): roles background_task = webhook_dispatcher roles = RoleSet { roles: {} }
28152025-05-08T22:03:38.362ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database result = Ok(())
28162025-05-08T22:03:38.368ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
28172025-05-08T22:03:38.368ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
28182025-05-08T22:03:38.370ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28192025-05-08T22:03:38.374ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
28202025-05-08T22:03:38.374ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
28212025-05-08T22:03:38.374ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
28222025-05-08T22:03:38.374ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
28232025-05-08T22:03:38.374ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
28242025-05-08T22:03:38.374ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
28252025-05-08T22:03:38.374ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
28262025-05-08T22:03:38.377ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:362
28272025-05-08T22:03:38.377ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 107.676042ms iteration = 11 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
28282025-05-08T22:03:38.377ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
28292025-05-08T22:03:38.377ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
28302025-05-08T22:03:38.377ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28312025-05-08T22:03:38.378ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
28322025-05-08T22:03:38.379ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
28332025-05-08T22:03:38.379ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
28342025-05-08T22:03:38.379ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
28352025-05-08T22:03:38.380ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
28362025-05-08T22:03:38.380ZDEBGnexus (BackgroundTasks): activation complete background_task = webhook_dispatcher elapsed = 96.712644ms iteration = 6 status = {"dispatched":[],"errors":[],"glob_version":"140.0.0","globs_reprocessed":{},"no_receivers":[]}
28372025-05-08T22:03:38.380ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 96.493802ms iteration = 11 status = {"errors":[],"requests_created_ok":[]}
28382025-05-08T22:03:38.381ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
28392025-05-08T22:03:38.381ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
28402025-05-08T22:03:38.384ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 103.971792ms iteration = 31 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
28412025-05-08T22:03:38.387ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:552 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
28422025-05-08T22:03:38.387ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 106.827932ms iteration = 11 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
28432025-05-08T22:03:38.390ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
28442025-05-08T22:03:38.390ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
28452025-05-08T22:03:38.393ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
28462025-05-08T22:03:38.393ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
28472025-05-08T22:03:38.394ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
28482025-05-08T22:03:38.394ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
28492025-05-08T22:03:38.394ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
28502025-05-08T22:03:38.394ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
28512025-05-08T22:03:38.395ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
28522025-05-08T22:03:38.395ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:137 found = 0 not_ready_to_be_deleted = 0
28532025-05-08T22:03:38.395ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 114.180681ms iteration = 6 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
28542025-05-08T22:03:38.395ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 7586992a-6fc5-4073-8962-728778f16aa5 dns_zone_name = control-plane.oxide.internal found_so_far = 55 version = 1
28552025-05-08T22:03:38.395ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
28562025-05-08T22:03:38.395ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-05-08 21:59:21.182334 UTC dns_group = internal generation = 1
28572025-05-08T22:03:38.395ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 128.790627ms iteration = 6 status = {"generation":1}
28582025-05-08T22:03:38.401ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28592025-05-08T22:03:38.401ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
28602025-05-08T22:03:38.401ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
28612025-05-08T22:03:38.402ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28622025-05-08T22:03:38.402ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
28632025-05-08T22:03:38.402ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = d8128696-db5a-4b88-914a-69b28494e50e dns_zone_name = oxide.test found_so_far = 0 version = 2
28642025-05-08T22:03:38.402ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
28652025-05-08T22:03:38.403ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
28662025-05-08T22:03:38.404ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28672025-05-08T22:03:38.404ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
28682025-05-08T22:03:38.404ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
28692025-05-08T22:03:38.404ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28702025-05-08T22:03:38.405ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
28712025-05-08T22:03:38.405ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
28722025-05-08T22:03:38.405ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
28732025-05-08T22:03:38.406ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
28742025-05-08T22:03:38.406ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
28752025-05-08T22:03:38.406ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
28762025-05-08T22:03:38.407ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "c72a647a-a6e3-46f9-9fbe-c5c5d46d3606", "content-length": "26", "date": "Thu, 08 May 2025 22:03:38 GMT"} })
28772025-05-08T22:03:38.407ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
28782025-05-08T22:03:38.408ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
28792025-05-08T22:03:38.408ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
28802025-05-08T22:03:38.409ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
28812025-05-08T22:03:38.410ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
28822025-05-08T22:03:38.410ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
28832025-05-08T22:03:38.410ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
28842025-05-08T22:03:38.410ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
28852025-05-08T22:03:38.412ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28862025-05-08T22:03:38.412ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
28872025-05-08T22:03:38.413ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
28882025-05-08T22:03:38.415ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
28892025-05-08T22:03:38.415ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
28902025-05-08T22:03:38.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
28912025-05-08T22:03:38.415ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
28922025-05-08T22:03:38.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
28932025-05-08T22:03:38.417ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28942025-05-08T22:03:38.422ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
28952025-05-08T22:03:38.422ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
28962025-05-08T22:03:38.422ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
28972025-05-08T22:03:38.424ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
28982025-05-08T22:03:38.424ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
28992025-05-08T22:03:38.425ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
29002025-05-08T22:03:38.425ZDEBGnexus (BackgroundTasks): activation complete background_task = webhook_deliverator elapsed = 141.928787ms iteration = 6 status = {"by_rx":{},"error":null}
29012025-05-08T22:03:38.426ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
29022025-05-08T22:03:38.427ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
29032025-05-08T22:03:38.429ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
29042025-05-08T22:03:38.430ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
29052025-05-08T22:03:38.432ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
29062025-05-08T22:03:38.432ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
29072025-05-08T22:03:38.432ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
29082025-05-08T22:03:38.433ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
29092025-05-08T22:03:38.436ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29102025-05-08T22:03:38.439ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
29112025-05-08T22:03:38.439ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
29122025-05-08T22:03:38.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
29132025-05-08T22:03:38.441ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
29142025-05-08T22:03:38.441ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
29152025-05-08T22:03:38.447ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
29162025-05-08T22:03:38.447ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
29172025-05-08T22:03:38.452ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 169.118238ms iteration = 11 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
29182025-05-08T22:03:38.452ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
29192025-05-08T22:03:38.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
29202025-05-08T22:03:38.455ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
29212025-05-08T22:03:38.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
29222025-05-08T22:03:38.457ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29232025-05-08T22:03:38.457ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29242025-05-08T22:03:38.458ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
29252025-05-08T22:03:38.458ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
29262025-05-08T22:03:38.458ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 177.67582ms iteration = 11 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
29272025-05-08T22:03:38.460ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
29282025-05-08T22:03:38.460ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
29292025-05-08T22:03:38.460ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
29302025-05-08T22:03:38.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
29312025-05-08T22:03:38.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
29322025-05-08T22:03:38.460ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-05-08T21:58:35.560911Z, time_modified: 2025-05-08T21:58:35.560911Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, db_silo: Silo { identity: SiloIdentity { id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-05-08T21:59:21.709488Z, time_modified: 2025-05-08T21:59:21.709488Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, db_silo: Silo { identity: SiloIdentity { id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-05-08T21:59:21.709488Z, time_modified: 2025-05-08T21:59:21.709488Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47" }] }) }
29332025-05-08T22:03:38.460ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 194.429459ms iteration = 6 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"7137ee58-6aa7-4199-a248-feadf3b8bafb","tls_certs":["37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47"]}},"default_endpoint":{"silo_id":"7137ee58-6aa7-4199-a248-feadf3b8bafb","tls_certs":["37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
29342025-05-08T22:03:38.460ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
29352025-05-08T22:03:38.460ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
29362025-05-08T22:03:38.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
29372025-05-08T22:03:38.464ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = d8128696-db5a-4b88-914a-69b28494e50e dns_zone_name = oxide.test found_so_far = 1 version = 2
29382025-05-08T22:03:38.464ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
29392025-05-08T22:03:38.464ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-05-08 21:59:21.759368 UTC dns_group = external generation = 2
29402025-05-08T22:03:38.464ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 198.359962ms iteration = 6 status = {"generation":2}
29412025-05-08T22:03:38.465ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
29422025-05-08T22:03:38.465ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 198.148969ms iteration = 31 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
29432025-05-08T22:03:38.466ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
29442025-05-08T22:03:38.467ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
29452025-05-08T22:03:38.468ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29462025-05-08T22:03:38.470ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
29472025-05-08T22:03:38.470ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
29482025-05-08T22:03:38.470ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
29492025-05-08T22:03:38.474ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
29502025-05-08T22:03:38.474ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
29512025-05-08T22:03:38.476ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29522025-05-08T22:03:38.478ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
29532025-05-08T22:03:38.478ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
29542025-05-08T22:03:38.479ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
29552025-05-08T22:03:38.482ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
29562025-05-08T22:03:38.483ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
29572025-05-08T22:03:38.484ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
29582025-05-08T22:03:38.485ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
29592025-05-08T22:03:38.501ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 218.068042ms iteration = 11 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
29602025-05-08T22:03:38.522ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 242.061598ms iteration = 12 status = {"physical_disks_added":0}
29612025-05-08T22:03:38.584ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
29622025-05-08T22:03:38.585ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
29632025-05-08T22:03:38.585ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
29642025-05-08T22:03:38.585ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
29652025-05-08T22:03:38.604ZINFOnexus (BackgroundTasks): instance reincarnation completed successfully background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:113 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0
29662025-05-08T22:03:38.604ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 323.587201ms iteration = 6 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
29672025-05-08T22:03:38.617ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
29682025-05-08T22:03:38.651ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 385.321021ms iteration = 11 status = {}
29692025-05-08T22:03:38.691ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
29702025-05-08T22:03:38.691ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
29712025-05-08T22:03:38.691ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c32ac17-1643-4d74-9104-8f9a17e67f59", "content-length": "26", "date": "Thu, 08 May 2025 22:03:38 GMT"} })
29722025-05-08T22:03:38.691ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
29732025-05-08T22:03:38.693ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
29742025-05-08T22:03:38.736ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
29752025-05-08T22:03:38.736ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
29762025-05-08T22:03:38.750ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e zone_address = fd00:1122:3344:101::2
29772025-05-08T22:03:38.750ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
29782025-05-08T22:03:38.751ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "16e82a3c-5555-4bb2-832f-40098f9e2b87", "content-length": "26", "date": "Thu, 08 May 2025 22:03:38 GMT"} })
29792025-05-08T22:03:38.751ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1174 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
29802025-05-08T22:03:38.753ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
29812025-05-08T22:03:38.759ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
29822025-05-08T22:03:38.818ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
29832025-05-08T22:03:38.818ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
29842025-05-08T22:03:38.822ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29852025-05-08T22:03:38.822ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29862025-05-08T22:03:38.826ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29872025-05-08T22:03:38.827ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
29882025-05-08T22:03:38.827ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
29892025-05-08T22:03:38.827ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
29902025-05-08T22:03:38.827ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
29912025-05-08T22:03:38.828ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
29922025-05-08T22:03:38.828ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
29932025-05-08T22:03:38.828ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
29942025-05-08T22:03:38.829ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
29952025-05-08T22:03:38.829ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
29962025-05-08T22:03:38.829ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
29972025-05-08T22:03:38.829ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
29982025-05-08T22:03:38.829ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29992025-05-08T22:03:38.829ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30002025-05-08T22:03:38.831ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
30012025-05-08T22:03:38.831ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
30022025-05-08T22:03:38.832ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
30032025-05-08T22:03:38.832ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
30042025-05-08T22:03:38.832ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
30052025-05-08T22:03:38.832ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
30062025-05-08T22:03:38.832ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
30072025-05-08T22:03:38.833ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30082025-05-08T22:03:38.834ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 2f2f2c32-8d58-4801-a669-9618eac0c3f7, lookup_type: ById(2f2f2c32-8d58-4801-a669-9618eac0c3f7) } result = Ok(())
30092025-05-08T22:03:38.834ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
30102025-05-08T22:03:38.835ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
30112025-05-08T22:03:38.835ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
30122025-05-08T22:03:38.835ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
30132025-05-08T22:03:38.836ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
30142025-05-08T22:03:38.836ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
30152025-05-08T22:03:38.836ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30162025-05-08T22:03:38.836ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
30172025-05-08T22:03:38.837ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
30182025-05-08T22:03:38.837ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
30192025-05-08T22:03:38.838ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
30202025-05-08T22:03:38.838ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
30212025-05-08T22:03:38.838ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 557.175689ms iteration = 11 status = 0
30222025-05-08T22:03:38.838ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
30232025-05-08T22:03:38.838ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
30242025-05-08T22:03:38.839ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
30252025-05-08T22:03:38.839ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30262025-05-08T22:03:38.840ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
30272025-05-08T22:03:38.840ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
30282025-05-08T22:03:38.841ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
30292025-05-08T22:03:38.841ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
30302025-05-08T22:03:38.841ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
30312025-05-08T22:03:38.841ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
30322025-05-08T22:03:38.841ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
30332025-05-08T22:03:38.841ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30342025-05-08T22:03:38.844ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
30352025-05-08T22:03:38.844ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
30362025-05-08T22:03:38.844ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
30372025-05-08T22:03:38.859ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
30382025-05-08T22:03:38.860ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "c6ee4699-aa3c-4bca-be8c-45f3780d3fe1", "date": "Thu, 08 May 2025 22:03:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
30392025-05-08T22:03:38.860ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
30402025-05-08T22:03:38.861ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "0ca01f15-bc5a-4877-b809-24290f6d56de", "date": "Thu, 08 May 2025 22:03:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
30412025-05-08T22:03:38.861ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
30422025-05-08T22:03:38.862ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
30432025-05-08T22:03:38.862ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 596.238408ms iteration = 8 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
30442025-05-08T22:03:38.937ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
30452025-05-08T22:03:38.938ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
30462025-05-08T22:03:38.940ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "1be9f946-d025-4f41-b4e1-cfeb62e46b5e", "content-length": "372", "date": "Thu, 08 May 2025 22:03:38 GMT"} })
30472025-05-08T22:03:38.940ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
30482025-05-08T22:03:38.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
30492025-05-08T22:03:39.000ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
30502025-05-08T22:03:39.000ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "6064ef1f-475c-4e2a-906f-4784a43ee3ca", "date": "Thu, 08 May 2025 22:03:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
30512025-05-08T22:03:39.000ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
30522025-05-08T22:03:39.002ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "0afca920-61b6-43c7-bfa5-43db2c13294f", "date": "Thu, 08 May 2025 22:03:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
30532025-05-08T22:03:39.002ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
30542025-05-08T22:03:39.003ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "d0a40c3d-8ae5-4677-b0a8-fa8791945d6d", "date": "Thu, 08 May 2025 22:03:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
30552025-05-08T22:03:39.003ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
30562025-05-08T22:03:39.005ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
30572025-05-08T22:03:39.005ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 739.173976ms iteration = 8 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
30582025-05-08T22:03:39.014ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
30592025-05-08T22:03:39.014ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9ab8381-acb2-4e5b-83bc-f97c47a115a1", "content-length": "691", "date": "Thu, 08 May 2025 22:03:38 GMT"} })
30602025-05-08T22:03:39.014ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
30612025-05-08T22:03:39.014ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
30622025-05-08T22:03:39.014ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
30632025-05-08T22:03:39.016ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
30642025-05-08T22:03:39.016ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 735.563188ms iteration = 11 status = {}
30652025-05-08T22:03:39.016ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
30662025-05-08T22:03:39.016ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
30672025-05-08T22:03:39.016ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = tuf_artifact_replication body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/artifacts-config
30682025-05-08T22:03:39.017ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
30692025-05-08T22:03:39.017ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = tuf_artifact_replication result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/artifacts-config", status: 204, headers: {"x-request-id": "051d80b7-8d0d-4783-9219-ac49c9a37275", "date": "Thu, 08 May 2025 22:03:38 GMT"} })
30702025-05-08T22:03:39.018ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 751.72039ms iteration = 11 status = null
30712025-05-08T22:03:39.018ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
30722025-05-08T22:03:39.018ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30732025-05-08T22:03:39.018ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
30742025-05-08T22:03:39.019ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = tuf_artifact_replication body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/artifacts
30752025-05-08T22:03:39.020ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = tuf_artifact_replication result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/artifacts", status: 200, headers: {"content-type": "application/json", "x-request-id": "213bda53-3168-49ad-ab7e-ce4c78fad938", "content-length": "26", "date": "Thu, 08 May 2025 22:03:38 GMT"} })
30762025-05-08T22:03:39.020ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30772025-05-08T22:03:39.020ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
30782025-05-08T22:03:39.020ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
30792025-05-08T22:03:39.020ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
30802025-05-08T22:03:39.022ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
30812025-05-08T22:03:39.022ZINFOnexus (BackgroundTasks): Successfully got artifact list background_task = tuf_artifact_replication file = nexus/src/app/background/tasks/tuf_artifact_replication.rs:673 sled = http://[fd00:1122:3344:101::1]:12345
30822025-05-08T22:03:39.022ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
30832025-05-08T22:03:39.024ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
30842025-05-08T22:03:39.024ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
30852025-05-08T22:03:39.024ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30862025-05-08T22:03:39.024ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
30872025-05-08T22:03:39.024ZDEBGnexus (BackgroundTasks): activation complete background_task = tuf_artifact_replication elapsed = 740.793054ms iteration = 2 status = {"generation":1,"last_run_counters":{"copy_err":0,"copy_ok":0,"list_err":0,"list_ok":1,"put_config_err":0,"put_config_ok":1,"put_err":0,"put_ok":0},"lifetime_counters":{"copy_err":0,"copy_ok":0,"list_err":0,"list_ok":1,"put_config_err":0,"put_config_ok":1,"put_err":0,"put_ok":0},"local_repos":0,"request_debug_ringbuf":[{"generation":1,"operation":"put_config","target_sled":"2f2f2c32-8d58-4801-a669-9618eac0c3f7","time":"2025-05-08T22:03:39.017442287Z"},{"operation":"list","target_sled":"2f2f2c32-8d58-4801-a669-9618eac0c3f7","time":"2025-05-08T22:03:39.022391931Z"}]}
30882025-05-08T22:03:39.025ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
30892025-05-08T22:03:39.026ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
30902025-05-08T22:03:39.026ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
30912025-05-08T22:03:39.026ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
30922025-05-08T22:03:39.026ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
30932025-05-08T22:03:39.027ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
30942025-05-08T22:03:39.027ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
30952025-05-08T22:03:39.027ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
30962025-05-08T22:03:39.027ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
30972025-05-08T22:03:39.029ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30982025-05-08T22:03:39.029ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30992025-05-08T22:03:39.030ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
31002025-05-08T22:03:39.030ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
31012025-05-08T22:03:39.030ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "a47d2dac-1c70-4450-862e-da3c90c5c838", "date": "Thu, 08 May 2025 22:03:38 GMT"} })
31022025-05-08T22:03:39.030ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
31032025-05-08T22:03:39.031ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
31042025-05-08T22:03:39.031ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
31052025-05-08T22:03:39.031ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
31062025-05-08T22:03:39.031ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
31072025-05-08T22:03:39.032ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
31082025-05-08T22:03:39.032ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31092025-05-08T22:03:39.033ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
31102025-05-08T22:03:39.033ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
31112025-05-08T22:03:39.033ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
31122025-05-08T22:03:39.033ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
31132025-05-08T22:03:39.034ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
31142025-05-08T22:03:39.034ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
31152025-05-08T22:03:39.034ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
31162025-05-08T22:03:39.037ZINFOnexus (BackgroundTasks): ensured all Crucible datasets present in inventory have database records background_task = blueprint_rendezvous file = nexus/reconfigurator/rendezvous/src/crucible_dataset.rs:113 num_already_exist = 9 num_inserted = 0 num_not_in_inventory = 0
31172025-05-08T22:03:39.037ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 756.445821ms iteration = 3 status = {"blueprint_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","inventory_collection_id":"ce348c8a-5059-46b8-8fc3-41419986897e","stats":{"crucible_dataset":{"num_already_exist":9,"num_inserted":0,"num_not_in_inventory":0},"debug_dataset":{"num_already_exist":5,"num_already_tombstoned":0,"num_inserted":4,"num_not_in_inventory":0,"num_tombstoned":0}}}
31182025-05-08T22:03:39.038ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
31192025-05-08T22:03:39.038ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
31202025-05-08T22:03:39.038ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
31212025-05-08T22:03:39.038ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
31222025-05-08T22:03:39.039ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
31232025-05-08T22:03:39.039ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
31242025-05-08T22:03:39.040ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
31252025-05-08T22:03:39.040ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
31262025-05-08T22:03:39.040ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "194e9c8b-8cd3-4053-ac67-271f230097b7", "date": "Thu, 08 May 2025 22:03:38 GMT"} })
31272025-05-08T22:03:39.041ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
31282025-05-08T22:03:39.041ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
31292025-05-08T22:03:39.043ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 762.282713ms iteration = 11 status = {}
31302025-05-08T22:03:39.043ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
31312025-05-08T22:03:39.043ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
31322025-05-08T22:03:39.045ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
31332025-05-08T22:03:39.046ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
31342025-05-08T22:03:39.047ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
31352025-05-08T22:03:39.047ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
31362025-05-08T22:03:39.048ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
31372025-05-08T22:03:39.048ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
31382025-05-08T22:03:39.050ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31392025-05-08T22:03:39.053ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
31402025-05-08T22:03:39.053ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
31412025-05-08T22:03:39.053ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
31422025-05-08T22:03:39.055ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
31432025-05-08T22:03:39.055ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
31442025-05-08T22:03:39.056ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
31452025-05-08T22:03:39.056ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
31462025-05-08T22:03:39.058ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31472025-05-08T22:03:39.059ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
31482025-05-08T22:03:39.059ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
31492025-05-08T22:03:39.061ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: cd0b930c-534b-46ad-ac3b-2eaa94b0bf2c, rack_id: 64f3d6f4-89b8-48aa-a622-b684e261667e, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(65f83a36-18fb-49b7-9914-87c8517d200e) } rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
31502025-05-08T22:03:39.061ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
31512025-05-08T22:03:39.061ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
31522025-05-08T22:03:39.062ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
31532025-05-08T22:03:39.062ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
31542025-05-08T22:03:39.062ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
31552025-05-08T22:03:39.069ZDEBGnexus (service-firewall-rule-progator): resolved names for firewall rules background_task = service_firewall_rule_propagation instance_interfaces = {} subnet_interfaces = {Name("external-dns"): [NetworkInterface { id: 517232ae-3fd2-40fd-801d-796bf5ecd14e, kind: Service { id: 0ccabf41-81cd-45e7-bb26-335edaa2ee47 }, name: Name("external-dns-0ccabf41-81cd-45e7-bb26-335edaa2ee47"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 224, 221])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: b845b0e9-661e-4fa1-9292-0c7dce84b88d, kind: Service { id: 38a08dd6-6a75-4558-b4f8-df853f4d03d9 }, name: Name("external-dns-38a08dd6-6a75-4558-b4f8-df853f4d03d9"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 179, 43])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], Name("nexus"): [NetworkInterface { id: 559b827c-9574-475a-93d6-1045f64398e6, kind: Service { id: 85a39cf9-9419-458f-af2e-c8b40d8f5f79 }, name: Name("nexus-85a39cf9-9419-458f-af2e-c8b40d8f5f79"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 232, 9])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 715a8bac-a423-4460-aac2-b8c76af6bb34, kind: Service { id: ddd82634-2844-4f97-9746-07099044c19d }, name: Name("nexus-ddd82634-2844-4f97-9746-07099044c19d"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 227, 84])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 76144c16-58bd-4abd-b758-565497d9f6b8, kind: Service { id: 53ef6679-65e8-4486-9401-d6fa4ff2e8e4 }, name: Name("nexus-53ef6679-65e8-4486-9401-d6fa4ff2e8e4"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 174, 62])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }]} subnet_networks = {Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })], Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })]} vpc_interfaces = {}
31562025-05-08T22:03:39.069ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
31572025-05-08T22:03:39.069ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
31582025-05-08T22:03:39.070ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31592025-05-08T22:03:39.072ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1833 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
31602025-05-08T22:03:39.072ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1860 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 65f83a36-18fb-49b7-9914-87c8517d200e
31612025-05-08T22:03:39.072ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
31622025-05-08T22:03:39.072ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
31632025-05-08T22:03:39.072ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
31642025-05-08T22:03:39.072ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
31652025-05-08T22:03:39.072ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
31662025-05-08T22:03:39.072ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2007 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
31672025-05-08T22:03:39.072ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Fleet result = Ok(())
31682025-05-08T22:03:39.072ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
31692025-05-08T22:03:39.072ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
31702025-05-08T22:03:39.072ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
31712025-05-08T22:03:39.072ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2043 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
31722025-05-08T22:03:39.072ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
31732025-05-08T22:03:39.072ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
31742025-05-08T22:03:39.073ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
31752025-05-08T22:03:39.074ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
31762025-05-08T22:03:39.074ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
31772025-05-08T22:03:39.184ZDEBGnexus (service-firewall-rule-progator): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified. background_task = service_firewall_rule_propagation
31782025-05-08T22:03:39.184ZDEBGnexus (service-firewall-rule-progator): resolved firewall rules for sled agents background_task = service_firewall_rule_propagation sled_agent_rules = [ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("53")]), filter_protocols: Some([Udp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 517232ae-3fd2-40fd-801d-796bf5ecd14e, kind: Service { id: 0ccabf41-81cd-45e7-bb26-335edaa2ee47 }, name: Name("external-dns-0ccabf41-81cd-45e7-bb26-335edaa2ee47"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 224, 221])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: b845b0e9-661e-4fa1-9292-0c7dce84b88d, kind: Service { id: 38a08dd6-6a75-4558-b4f8-df853f4d03d9 }, name: Name("external-dns-38a08dd6-6a75-4558-b4f8-df853f4d03d9"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 179, 43])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }, ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("80"), L4PortRange("443")]), filter_protocols: Some([Tcp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 559b827c-9574-475a-93d6-1045f64398e6, kind: Service { id: 85a39cf9-9419-458f-af2e-c8b40d8f5f79 }, name: Name("nexus-85a39cf9-9419-458f-af2e-c8b40d8f5f79"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 232, 9])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 715a8bac-a423-4460-aac2-b8c76af6bb34, kind: Service { id: ddd82634-2844-4f97-9746-07099044c19d }, name: Name("nexus-ddd82634-2844-4f97-9746-07099044c19d"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 227, 84])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 76144c16-58bd-4abd-b758-565497d9f6b8, kind: Service { id: 53ef6679-65e8-4486-9401-d6fa4ff2e8e4 }, name: Name("nexus-53ef6679-65e8-4486-9401-d6fa4ff2e8e4"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 174, 62])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }]
31792025-05-08T22:03:39.184ZDEBGnexus (service-firewall-rule-progator): resolved 2 rules for sleds background_task = service_firewall_rule_propagation
31802025-05-08T22:03:39.198ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "24af1900-4521-4fdb-8b97-01daff4e6f4d", "date": "Thu, 08 May 2025 22:03:38 GMT"} })
31812025-05-08T22:03:39.198ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch = Switch0
31822025-05-08T22:03:39.198ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
31832025-05-08T22:03:39.206ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "c31d8b9c-e95e-487c-b2f2-d52780a2300c", "content-length": "682", "date": "Thu, 08 May 2025 22:03:39 GMT"} })
31842025-05-08T22:03:39.206ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
31852025-05-08T22:03:39.206ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
31862025-05-08T22:03:39.211ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
31872025-05-08T22:03:39.212ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
31882025-05-08T22:03:39.270ZDEBGnexus (service-firewall-rule-progator): resolved sleds for vpc oxide-services background_task = service_firewall_rule_propagation vpc_to_sled = [Sled { identity: SledIdentity { id: 2f2f2c32-8d58-4801-a669-9618eac0c3f7, time_created: 2025-05-08T21:59:12.125305Z, time_modified: 2025-05-08T21:59:12.125305Z }, time_deleted: None, rcgen: Generation(Generation(19)), rack_id: 64f3d6f4-89b8-48aa-a622-b684e261667e, is_scrimlet: true, serial_number: "sock", part_number: "i86pc", revision: SqlU32(0), usable_hardware_threads: SqlU32(48), usable_physical_ram: ByteCount(ByteCount(102952026112)), reservoir_size: ByteCount(ByteCount(56094621696)), ip: fd00:1122:3344:101::1, port: SqlU16(12345), last_used_address: fd00:1122:3344:101::ffff, policy: InService, state: Active, sled_agent_gen: Generation(Generation(1)), repo_depot_port: SqlU16(12348) }]
31892025-05-08T22:03:39.270ZDEBGnexus (service-firewall-rule-progator): sending firewall rules to sled agents background_task = service_firewall_rule_propagation
31902025-05-08T22:03:39.270ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
31912025-05-08T22:03:39.270ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
31922025-05-08T22:03:39.272ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
31932025-05-08T22:03:39.272ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
31942025-05-08T22:03:39.273ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31952025-05-08T22:03:39.276ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Sled { parent: Fleet, key: 2f2f2c32-8d58-4801-a669-9618eac0c3f7, lookup_type: ById(2f2f2c32-8d58-4801-a669-9618eac0c3f7) } result = Ok(())
31962025-05-08T22:03:39.307ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
31972025-05-08T22:03:39.307ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
31982025-05-08T22:03:39.308ZDEBGnexus (service-firewall-rule-progator): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = service_firewall_rule_propagation body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules
31992025-05-08T22:03:39.308ZDEBGnexus (service-firewall-rule-progator): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = service_firewall_rule_propagation result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules", status: 204, headers: {"x-request-id": "6a1c7e33-0e21-4fef-9911-77d783b98679", "date": "Thu, 08 May 2025 22:03:38 GMT"} })
32002025-05-08T22:03:39.309ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
32012025-05-08T22:03:39.309ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
32022025-05-08T22:03:39.311ZINFOnexus (service-firewall-rule-progator): updated firewall rules on 1 sleds background_task = service_firewall_rule_propagation file = nexus/networking/src/firewall_rules.rs:495
32032025-05-08T22:03:39.311ZDEBGnexus (service-firewall-rule-progator): successfully propagated service firewall rules background_task = service_firewall_rule_propagation elapsed = 1.030212959s
32042025-05-08T22:03:39.311ZDEBGnexus (BackgroundTasks): activation complete background_task = service_firewall_rule_propagation elapsed = 1.03022384s iteration = 2 status = {}
32052025-05-08T22:03:39.316ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 1.03592129s iteration = 11 status = {}
32062025-05-08T22:03:41.935ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 287 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 9bdc38ba-856c-4a55-9917-c341d983936c response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
32072025-05-08T22:03:48.267ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 32 reason = Timeout
32082025-05-08T22:03:48.267ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32092025-05-08T22:03:48.268ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
32102025-05-08T22:03:48.270ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32112025-05-08T22:03:48.273ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
32122025-05-08T22:03:48.273ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32132025-05-08T22:03:48.273ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
32142025-05-08T22:03:48.274ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32152025-05-08T22:03:48.275ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
32162025-05-08T22:03:48.276ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32172025-05-08T22:03:48.279ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
32182025-05-08T22:03:48.279ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32192025-05-08T22:03:48.279ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
32202025-05-08T22:03:48.281ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 32 reason = Timeout
32212025-05-08T22:03:48.281ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
32222025-05-08T22:03:48.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
32232025-05-08T22:03:48.282ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
32242025-05-08T22:03:48.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
32252025-05-08T22:03:48.284ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.052233ms iteration = 32 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
32262025-05-08T22:03:48.288ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
32272025-05-08T22:03:48.288ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.788592ms iteration = 32 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
32282025-05-08T22:03:51.936ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 270 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 6a2d54f2-9376-4b1f-8dab-fd9fad91513b response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
32292025-05-08T22:03:58.268ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 33 reason = Timeout
32302025-05-08T22:03:58.268ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32312025-05-08T22:03:58.268ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
32322025-05-08T22:03:58.270ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32332025-05-08T22:03:58.273ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
32342025-05-08T22:03:58.273ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32352025-05-08T22:03:58.273ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
32362025-05-08T22:03:58.275ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32372025-05-08T22:03:58.275ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
32382025-05-08T22:03:58.277ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32392025-05-08T22:03:58.280ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
32402025-05-08T22:03:58.280ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32412025-05-08T22:03:58.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
32422025-05-08T22:03:58.280ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 33 reason = Timeout
32432025-05-08T22:03:58.280ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
32442025-05-08T22:03:58.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
32452025-05-08T22:03:58.282ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
32462025-05-08T22:03:58.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
32472025-05-08T22:03:58.284ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.514368ms iteration = 33 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
32482025-05-08T22:03:58.288ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
32492025-05-08T22:03:58.288ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.54902ms iteration = 33 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
32502025-05-08T22:04:01.936ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 304 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = c9c9fc86-ad58-426e-94b5-c1273e1e9b77 response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
32512025-05-08T22:04:08.266ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 12 reason = Timeout
32522025-05-08T22:04:08.266ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
32532025-05-08T22:04:08.266ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 12 reason = Timeout
32542025-05-08T22:04:08.266ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
32552025-05-08T22:04:08.266ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
32562025-05-08T22:04:08.267ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
32572025-05-08T22:04:08.267ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 12 reason = Timeout
32582025-05-08T22:04:08.267ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 34 reason = Timeout
32592025-05-08T22:04:08.267ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32602025-05-08T22:04:08.268ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
32612025-05-08T22:04:08.269ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1224, tv_nsec: 10493141 } })
32622025-05-08T22:04:08.269ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1224, tv_nsec: 10729773 } })
32632025-05-08T22:04:08.269ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 12 reason = Timeout
32642025-05-08T22:04:08.269ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
32652025-05-08T22:04:08.269ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1163
32662025-05-08T22:04:08.269ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163
32672025-05-08T22:04:08.270ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
32682025-05-08T22:04:08.270ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32692025-05-08T22:04:08.271ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.581369ms iteration = 12 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
32702025-05-08T22:04:08.271ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32712025-05-08T22:04:08.273ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
32722025-05-08T22:04:08.273ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32732025-05-08T22:04:08.274ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
32742025-05-08T22:04:08.274ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
32752025-05-08T22:04:08.274ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
32762025-05-08T22:04:08.274ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
32772025-05-08T22:04:08.276ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32782025-05-08T22:04:08.276ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
32792025-05-08T22:04:08.277ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32802025-05-08T22:04:08.280ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
32812025-05-08T22:04:08.280ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32822025-05-08T22:04:08.280ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 34 reason = Timeout
32832025-05-08T22:04:08.280ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 12 reason = Timeout
32842025-05-08T22:04:08.280ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
32852025-05-08T22:04:08.280ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 12 reason = Timeout
32862025-05-08T22:04:08.280ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
32872025-05-08T22:04:08.280ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
32882025-05-08T22:04:08.280ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 12 reason = Timeout
32892025-05-08T22:04:08.280ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
32902025-05-08T22:04:08.280ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
32912025-05-08T22:04:08.280ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 12 reason = Timeout
32922025-05-08T22:04:08.280ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
32932025-05-08T22:04:08.280ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 12 reason = Timeout
32942025-05-08T22:04:08.280ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
32952025-05-08T22:04:08.280ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 12 reason = Timeout
32962025-05-08T22:04:08.280ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
32972025-05-08T22:04:08.280ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 13 reason = Timeout
32982025-05-08T22:04:08.280ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 12 reason = Timeout
32992025-05-08T22:04:08.280ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
33002025-05-08T22:04:08.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
33012025-05-08T22:04:08.280ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
33022025-05-08T22:04:08.280ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
33032025-05-08T22:04:08.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
33042025-05-08T22:04:08.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
33052025-05-08T22:04:08.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
33062025-05-08T22:04:08.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
33072025-05-08T22:04:08.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
33082025-05-08T22:04:08.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
33092025-05-08T22:04:08.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
33102025-05-08T22:04:08.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
33112025-05-08T22:04:08.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
33122025-05-08T22:04:08.282ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
33132025-05-08T22:04:08.282ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 12 reason = Timeout
33142025-05-08T22:04:08.282ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
33152025-05-08T22:04:08.282ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 12 reason = Timeout
33162025-05-08T22:04:08.282ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
33172025-05-08T22:04:08.282ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 12 reason = Timeout
33182025-05-08T22:04:08.282ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
33192025-05-08T22:04:08.283ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 12 reason = Timeout
33202025-05-08T22:04:08.283ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
33212025-05-08T22:04:08.283ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 12 reason = Timeout
33222025-05-08T22:04:08.283ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
33232025-05-08T22:04:08.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
33242025-05-08T22:04:08.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
33252025-05-08T22:04:08.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
33262025-05-08T22:04:08.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
33272025-05-08T22:04:08.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
33282025-05-08T22:04:08.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
33292025-05-08T22:04:08.286ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33302025-05-08T22:04:08.287ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33312025-05-08T22:04:08.287ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33322025-05-08T22:04:08.291ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
33332025-05-08T22:04:08.291ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
33342025-05-08T22:04:08.291ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
33352025-05-08T22:04:08.291ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
33362025-05-08T22:04:08.291ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
33372025-05-08T22:04:08.291ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
33382025-05-08T22:04:08.292ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
33392025-05-08T22:04:08.292ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
33402025-05-08T22:04:08.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
33412025-05-08T22:04:08.295ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
33422025-05-08T22:04:08.295ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 27.488374ms iteration = 34 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
33432025-05-08T22:04:08.296ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33442025-05-08T22:04:08.296ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 15.762258ms iteration = 34 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
33452025-05-08T22:04:08.298ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
33462025-05-08T22:04:08.298ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
33472025-05-08T22:04:08.299ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
33482025-05-08T22:04:08.299ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
33492025-05-08T22:04:08.299ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
33502025-05-08T22:04:08.301ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
33512025-05-08T22:04:08.301ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 18.401817ms iteration = 12 status = {"errors":[],"finish_invoked_ok":[]}
33522025-05-08T22:04:08.301ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
33532025-05-08T22:04:08.302ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1224, tv_nsec: 43449794 } })
33542025-05-08T22:04:08.302ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
33552025-05-08T22:04:08.302ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33562025-05-08T22:04:08.303ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
33572025-05-08T22:04:08.303ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
33582025-05-08T22:04:08.304ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 21.083146ms iteration = 12 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
33592025-05-08T22:04:08.304ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
33602025-05-08T22:04:08.304ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
33612025-05-08T22:04:08.306ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
33622025-05-08T22:04:08.306ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
33632025-05-08T22:04:08.306ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
33642025-05-08T22:04:08.311ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33652025-05-08T22:04:08.313ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
33662025-05-08T22:04:08.313ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
33672025-05-08T22:04:08.313ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
33682025-05-08T22:04:08.333ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
33692025-05-08T22:04:08.334ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
33702025-05-08T22:04:08.334ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33712025-05-08T22:04:08.337ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
33722025-05-08T22:04:08.338ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
33732025-05-08T22:04:08.338ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
33742025-05-08T22:04:08.338ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
33752025-05-08T22:04:08.338ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
33762025-05-08T22:04:08.338ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
33772025-05-08T22:04:08.339ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
33782025-05-08T22:04:08.343ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 60.048593ms iteration = 12 status = {"errors":[],"requests_created_ok":[]}
33792025-05-08T22:04:08.346ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
33802025-05-08T22:04:08.346ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
33812025-05-08T22:04:08.346ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
33822025-05-08T22:04:08.346ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
33832025-05-08T22:04:08.346ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
33842025-05-08T22:04:08.346ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
33852025-05-08T22:04:08.353ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 70.972079ms iteration = 12 status = {"errors":[],"garbage_collect_requested":[]}
33862025-05-08T22:04:08.354ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
33872025-05-08T22:04:08.354ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
33882025-05-08T22:04:08.354ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "90683e13-3007-4005-91e2-9bfa8442236c", "content-length": "26", "date": "Thu, 08 May 2025 22:04:08 GMT"} })
33892025-05-08T22:04:08.354ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
33902025-05-08T22:04:08.356ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
33912025-05-08T22:04:08.358ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
33922025-05-08T22:04:08.358ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
33932025-05-08T22:04:08.384ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
33942025-05-08T22:04:08.384ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
33952025-05-08T22:04:08.384ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ce35e47-f6bf-4886-8641-8b03652bc8f8", "content-length": "26", "date": "Thu, 08 May 2025 22:04:08 GMT"} })
33962025-05-08T22:04:08.384ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
33972025-05-08T22:04:08.386ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
33982025-05-08T22:04:08.467ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 201.370504ms iteration = 12 status = {}
33992025-05-08T22:04:08.500ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
34002025-05-08T22:04:08.500ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
34012025-05-08T22:04:08.504ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e zone_address = fd00:1122:3344:101::2
34022025-05-08T22:04:08.504ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
34032025-05-08T22:04:08.504ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f725a35-4a10-4564-914d-7ea7af5ed27e", "content-length": "26", "date": "Thu, 08 May 2025 22:04:08 GMT"} })
34042025-05-08T22:04:08.504ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1174 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
34052025-05-08T22:04:08.509ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
34062025-05-08T22:04:08.519ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
34072025-05-08T22:04:08.519ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
34082025-05-08T22:04:08.519ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b48f9e4-04f0-4f55-8b80-9cfb18046419", "content-length": "372", "date": "Thu, 08 May 2025 22:04:08 GMT"} })
34092025-05-08T22:04:08.519ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
34102025-05-08T22:04:08.520ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
34112025-05-08T22:04:08.523ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
34122025-05-08T22:04:08.523ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
34132025-05-08T22:04:08.525ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34142025-05-08T22:04:08.525ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
34152025-05-08T22:04:08.525ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
34162025-05-08T22:04:08.525ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
34172025-05-08T22:04:08.525ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
34182025-05-08T22:04:08.528ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
34192025-05-08T22:04:08.528ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
34202025-05-08T22:04:08.528ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
34212025-05-08T22:04:08.530ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 264.295526ms iteration = 12 status = null
34222025-05-08T22:04:08.531ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34232025-05-08T22:04:08.535ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 2f2f2c32-8d58-4801-a669-9618eac0c3f7, lookup_type: ById(2f2f2c32-8d58-4801-a669-9618eac0c3f7) } result = Ok(())
34242025-05-08T22:04:08.535ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
34252025-05-08T22:04:08.536ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
34262025-05-08T22:04:08.540ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 260.100752ms iteration = 12 status = 0
34272025-05-08T22:04:08.584ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 303.257514ms iteration = 12 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
34282025-05-08T22:04:08.584ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 301.897449ms iteration = 12 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
34292025-05-08T22:04:08.591ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:552 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
34302025-05-08T22:04:08.591ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 310.48412ms iteration = 12 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
34312025-05-08T22:04:08.591ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 310.46504ms iteration = 13 status = {"physical_disks_added":0}
34322025-05-08T22:04:08.599ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:362
34332025-05-08T22:04:08.599ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 329.802228ms iteration = 12 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
34342025-05-08T22:04:08.599ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
34352025-05-08T22:04:08.599ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
34362025-05-08T22:04:08.599ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 318.84621ms iteration = 12 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
34372025-05-08T22:04:08.602ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
34382025-05-08T22:04:08.602ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
34392025-05-08T22:04:08.604ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34402025-05-08T22:04:08.606ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
34412025-05-08T22:04:08.606ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
34422025-05-08T22:04:08.606ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
34432025-05-08T22:04:08.607ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34442025-05-08T22:04:08.609ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
34452025-05-08T22:04:08.609ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
34462025-05-08T22:04:08.609ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
34472025-05-08T22:04:08.620ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
34482025-05-08T22:04:08.620ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
34492025-05-08T22:04:08.620ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "4cdfa3ab-ced5-4557-bafe-4d273ad65e6c", "date": "Thu, 08 May 2025 22:04:08 GMT"} })
34502025-05-08T22:04:08.620ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
34512025-05-08T22:04:08.621ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
34522025-05-08T22:04:08.623ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
34532025-05-08T22:04:08.623ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
34542025-05-08T22:04:08.627ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
34552025-05-08T22:04:08.630ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
34562025-05-08T22:04:08.637ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
34572025-05-08T22:04:08.637ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
34582025-05-08T22:04:08.637ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "a54f3531-946a-4c0e-a96f-c815b5b7f6fd", "date": "Thu, 08 May 2025 22:04:08 GMT"} })
34592025-05-08T22:04:08.639ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 358.463014ms iteration = 12 status = {}
34602025-05-08T22:04:08.669ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
34612025-05-08T22:04:08.670ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
34622025-05-08T22:04:08.671ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: cd0b930c-534b-46ad-ac3b-2eaa94b0bf2c, rack_id: 64f3d6f4-89b8-48aa-a622-b684e261667e, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(65f83a36-18fb-49b7-9914-87c8517d200e) } rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
34632025-05-08T22:04:08.671ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
34642025-05-08T22:04:08.672ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
34652025-05-08T22:04:08.691ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1833 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
34662025-05-08T22:04:08.691ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1860 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 65f83a36-18fb-49b7-9914-87c8517d200e
34672025-05-08T22:04:08.692ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
34682025-05-08T22:04:08.692ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
34692025-05-08T22:04:08.692ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
34702025-05-08T22:04:08.692ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
34712025-05-08T22:04:08.692ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
34722025-05-08T22:04:08.692ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2007 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
34732025-05-08T22:04:08.692ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
34742025-05-08T22:04:08.692ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2043 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
34752025-05-08T22:04:08.692ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
34762025-05-08T22:04:08.692ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
34772025-05-08T22:04:08.692ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
34782025-05-08T22:04:08.695ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
34792025-05-08T22:04:08.695ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
34802025-05-08T22:04:08.723ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
34812025-05-08T22:04:08.798ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "bde7ce0f-496d-44a4-9e54-4506f97b0670", "date": "Thu, 08 May 2025 22:04:08 GMT"} })
34822025-05-08T22:04:08.798ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch = Switch0
34832025-05-08T22:04:08.798ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
34842025-05-08T22:04:08.799ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d48dc65-db59-47b8-93ff-29042361d0f3", "content-length": "682", "date": "Thu, 08 May 2025 22:04:08 GMT"} })
34852025-05-08T22:04:08.799ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
34862025-05-08T22:04:08.799ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
34872025-05-08T22:04:08.799ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "c445aacc-1d4b-40c3-bf33-f7d3a4c6c3c8", "content-length": "691", "date": "Thu, 08 May 2025 22:04:08 GMT"} })
34882025-05-08T22:04:08.799ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
34892025-05-08T22:04:08.799ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
34902025-05-08T22:04:08.799ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
34912025-05-08T22:04:08.800ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 519.80025ms iteration = 12 status = {}
34922025-05-08T22:04:08.802ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
34932025-05-08T22:04:08.803ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
34942025-05-08T22:04:08.806ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
34952025-05-08T22:04:08.807ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
34962025-05-08T22:04:08.809ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
34972025-05-08T22:04:08.809ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
34982025-05-08T22:04:08.815ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 534.87765ms iteration = 12 status = {}
34992025-05-08T22:04:11.935ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 262 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = a663016d-c7fe-4529-8220-70d57f934b42 response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
35002025-05-08T22:04:18.268ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 35 reason = Timeout
35012025-05-08T22:04:18.268ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
35022025-05-08T22:04:18.268ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
35032025-05-08T22:04:18.270ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35042025-05-08T22:04:18.272ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
35052025-05-08T22:04:18.272ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
35062025-05-08T22:04:18.272ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
35072025-05-08T22:04:18.276ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
35082025-05-08T22:04:18.277ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
35092025-05-08T22:04:18.280ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35102025-05-08T22:04:18.281ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 35 reason = Timeout
35112025-05-08T22:04:18.281ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
35122025-05-08T22:04:18.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
35132025-05-08T22:04:18.283ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
35142025-05-08T22:04:18.283ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
35152025-05-08T22:04:18.283ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
35162025-05-08T22:04:18.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
35172025-05-08T22:04:18.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
35182025-05-08T22:04:18.299ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
35192025-05-08T22:04:18.299ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 31.606248ms iteration = 35 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
35202025-05-08T22:04:18.315ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 34.893223ms iteration = 35 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
35212025-05-08T22:04:21.936ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 269 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 3ce7a9ca-f935-4852-8ad8-71a66bf40bb3 response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
35222025-05-08T22:04:28.268ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 36 reason = Timeout
35232025-05-08T22:04:28.268ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
35242025-05-08T22:04:28.269ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
35252025-05-08T22:04:28.270ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35262025-05-08T22:04:28.273ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
35272025-05-08T22:04:28.273ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
35282025-05-08T22:04:28.273ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
35292025-05-08T22:04:28.275ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
35302025-05-08T22:04:28.275ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
35312025-05-08T22:04:28.277ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35322025-05-08T22:04:28.280ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
35332025-05-08T22:04:28.280ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
35342025-05-08T22:04:28.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
35352025-05-08T22:04:28.281ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 36 reason = Timeout
35362025-05-08T22:04:28.281ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
35372025-05-08T22:04:28.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
35382025-05-08T22:04:28.290ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
35392025-05-08T22:04:28.290ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.161927ms iteration = 36 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
35402025-05-08T22:04:28.317ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
35412025-05-08T22:04:28.317ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
35422025-05-08T22:04:28.353ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 72.01684ms iteration = 36 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
35432025-05-08T22:04:31.936ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 413 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = c3b12a17-4dc3-46a8-a50e-e9efc47575fb response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
35442025-05-08T22:04:38.266ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 13 reason = Timeout
35452025-05-08T22:04:38.266ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
35462025-05-08T22:04:38.266ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 9 reason = Timeout
35472025-05-08T22:04:38.266ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 7 reason = Timeout
35482025-05-08T22:04:38.266ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
35492025-05-08T22:04:38.266ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
35502025-05-08T22:04:38.266ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 7 reason = Timeout
35512025-05-08T22:04:38.266ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
35522025-05-08T22:04:38.266ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 7 reason = Timeout
35532025-05-08T22:04:38.266ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 9 reason = Timeout
35542025-05-08T22:04:38.267ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 7 reason = Timeout
35552025-05-08T22:04:38.267ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 7 reason = Timeout
35562025-05-08T22:04:38.267ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 13 reason = Timeout
35572025-05-08T22:04:38.267ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
35582025-05-08T22:04:38.267ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
35592025-05-08T22:04:38.267ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
35602025-05-08T22:04:38.267ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 7 reason = Timeout
35612025-05-08T22:04:38.267ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-05-08 21:54:38.267057976 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
35622025-05-08T22:04:38.267ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
35632025-05-08T22:04:38.267ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
35642025-05-08T22:04:38.267ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
35652025-05-08T22:04:38.267ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
35662025-05-08T22:04:38.267ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("0ccabf41-81cd-45e7-bb26-335edaa2ee47.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("38a08dd6-6a75-4558-b4f8-df853f4d03d9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("38a08dd6-6a75-4558-b4f8-df853f4d03d9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::9))) }], valid_until: Instant { tv_sec: 1254, tv_nsec: 8211126 } })
35672025-05-08T22:04:38.267ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
35682025-05-08T22:04:38.267ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("228ed1c1-3cc1-4055-aae9-5595ad856b86.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("8b245480-f91d-406b-bb08-850e1c3e3fc5.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c8f20f67-ea63-4dbc-b0f1-c30876daff82.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c8f20f67-ea63-4dbc-b0f1-c30876daff82.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 1254, tv_nsec: 8282557 } })
35692025-05-08T22:04:38.267ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
35702025-05-08T22:04:38.267ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
35712025-05-08T22:04:38.267ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } dns_group = external
35722025-05-08T22:04:38.267ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 983.03\u{b5}s iteration = 7 status = {"addresses":["[fd00:1122:3344:101::8]:5353","[fd00:1122:3344:101::9]:5353"]}
35732025-05-08T22:04:38.267ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 37 reason = Timeout
35742025-05-08T22:04:38.267ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
35752025-05-08T22:04:38.267ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 8 reason = Timeout
35762025-05-08T22:04:38.267ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:85 reason = blueprint disabled target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2
35772025-05-08T22:04:38.267ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 13 reason = Timeout
35782025-05-08T22:04:38.267ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 9.01\u{b5}s iteration = 8 status = {"enabled":false,"target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2"}
35792025-05-08T22:04:38.268ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
35802025-05-08T22:04:38.268ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
35812025-05-08T22:04:38.268ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.134602ms iteration = 7 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:1::1]:5353"]}
35822025-05-08T22:04:38.268ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
35832025-05-08T22:04:38.268ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
35842025-05-08T22:04:38.270ZDEBGnexus (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 4 reason = Timeout
35852025-05-08T22:04:38.270ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 13 reason = Timeout
35862025-05-08T22:04:38.270ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
35872025-05-08T22:04:38.270ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
35882025-05-08T22:04:38.270ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35892025-05-08T22:04:38.270ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
35902025-05-08T22:04:38.270ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1254, tv_nsec: 11254049 } })
35912025-05-08T22:04:38.270ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
35922025-05-08T22:04:38.270ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
35932025-05-08T22:04:38.270ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
35942025-05-08T22:04:38.270ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
35952025-05-08T22:04:38.270ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
35962025-05-08T22:04:38.270ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1163
35972025-05-08T22:04:38.270ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
35982025-05-08T22:04:38.270ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
35992025-05-08T22:04:38.270ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
36002025-05-08T22:04:38.271ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
36012025-05-08T22:04:38.271ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
36022025-05-08T22:04:38.271ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
36032025-05-08T22:04:38.271ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-05-08 21:54:38.267057976 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
36042025-05-08T22:04:38.271ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 4.342477ms iteration = 7 status = {"expiration":"2025-05-08T21:54:38.267057976Z","pruned":[]}
36052025-05-08T22:04:38.271ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1254, tv_nsec: 12551693 } })
36062025-05-08T22:04:38.272ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163
36072025-05-08T22:04:38.273ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
36082025-05-08T22:04:38.273ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
36092025-05-08T22:04:38.273ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
36102025-05-08T22:04:38.275ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 7.156296ms iteration = 13 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
36112025-05-08T22:04:38.275ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36122025-05-08T22:04:38.275ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36132025-05-08T22:04:38.275ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36142025-05-08T22:04:38.277ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36152025-05-08T22:04:38.279ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
36162025-05-08T22:04:38.279ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
36172025-05-08T22:04:38.279ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
36182025-05-08T22:04:38.279ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
36192025-05-08T22:04:38.279ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
36202025-05-08T22:04:38.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
36212025-05-08T22:04:38.281ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36222025-05-08T22:04:38.281ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 13 reason = Timeout
36232025-05-08T22:04:38.281ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
36242025-05-08T22:04:38.281ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
36252025-05-08T22:04:38.281ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 7 reason = Timeout
36262025-05-08T22:04:38.281ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 13 reason = Timeout
36272025-05-08T22:04:38.281ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
36282025-05-08T22:04:38.281ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
36292025-05-08T22:04:38.281ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 13 reason = Timeout
36302025-05-08T22:04:38.281ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
36312025-05-08T22:04:38.281ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 7 reason = Timeout
36322025-05-08T22:04:38.281ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:53
36332025-05-08T22:04:38.281ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
36342025-05-08T22:04:38.281ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 14 reason = Timeout
36352025-05-08T22:04:38.281ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 13 reason = Timeout
36362025-05-08T22:04:38.281ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
36372025-05-08T22:04:38.281ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
36382025-05-08T22:04:38.281ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 13 reason = Timeout
36392025-05-08T22:04:38.281ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
36402025-05-08T22:04:38.281ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 37 reason = Timeout
36412025-05-08T22:04:38.281ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
36422025-05-08T22:04:38.281ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 13 reason = Timeout
36432025-05-08T22:04:38.281ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
36442025-05-08T22:04:38.281ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 13 reason = Timeout
36452025-05-08T22:04:38.281ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
36462025-05-08T22:04:38.281ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
36472025-05-08T22:04:38.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
36482025-05-08T22:04:38.281ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
36492025-05-08T22:04:38.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
36502025-05-08T22:04:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
36512025-05-08T22:04:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
36522025-05-08T22:04:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
36532025-05-08T22:04:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
36542025-05-08T22:04:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
36552025-05-08T22:04:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
36562025-05-08T22:04:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
36572025-05-08T22:04:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
36582025-05-08T22:04:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
36592025-05-08T22:04:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
36602025-05-08T22:04:38.283ZDEBGnexus (BackgroundTasks): activating background_task = webhook_deliverator iteration = 7 reason = Timeout
36612025-05-08T22:04:38.283ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 7 reason = Timeout
36622025-05-08T22:04:38.283ZDEBGnexus (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} }
36632025-05-08T22:04:38.283ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
36642025-05-08T22:04:38.283ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 13 reason = Timeout
36652025-05-08T22:04:38.283ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
36662025-05-08T22:04:38.283ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 13 reason = Timeout
36672025-05-08T22:04:38.283ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
36682025-05-08T22:04:38.283ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 13 reason = Timeout
36692025-05-08T22:04:38.283ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
36702025-05-08T22:04:38.283ZDEBGnexus (BackgroundTasks): activating background_task = webhook_dispatcher iteration = 7 reason = Timeout
36712025-05-08T22:04:38.283ZDEBGnexus (BackgroundTasks): roles background_task = webhook_dispatcher roles = RoleSet { roles: {} }
36722025-05-08T22:04:38.283ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 7 reason = Timeout
36732025-05-08T22:04:38.283ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
36742025-05-08T22:04:38.283ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 13 reason = Timeout
36752025-05-08T22:04:38.283ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
36762025-05-08T22:04:38.283ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 13 reason = Timeout
36772025-05-08T22:04:38.283ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
36782025-05-08T22:04:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
36792025-05-08T22:04:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
36802025-05-08T22:04:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
36812025-05-08T22:04:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
36822025-05-08T22:04:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
36832025-05-08T22:04:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database result = Ok(())
36842025-05-08T22:04:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
36852025-05-08T22:04:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database result = Ok(())
36862025-05-08T22:04:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
36872025-05-08T22:04:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
36882025-05-08T22:04:38.284ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
36892025-05-08T22:04:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
36902025-05-08T22:04:38.285ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36912025-05-08T22:04:38.287ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
36922025-05-08T22:04:38.288ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
36932025-05-08T22:04:38.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
36942025-05-08T22:04:38.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
36952025-05-08T22:04:38.289ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
36962025-05-08T22:04:38.289ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
36972025-05-08T22:04:38.289ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
36982025-05-08T22:04:38.326ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
36992025-05-08T22:04:38.326ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
37002025-05-08T22:04:38.326ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 43.175851ms iteration = 7 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
37012025-05-08T22:04:38.340ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37022025-05-08T22:04:38.344ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
37032025-05-08T22:04:38.344ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
37042025-05-08T22:04:38.344ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
37052025-05-08T22:04:38.355ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
37062025-05-08T22:04:38.355ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
37072025-05-08T22:04:38.355ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
37082025-05-08T22:04:38.355ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
37092025-05-08T22:04:38.355ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
37102025-05-08T22:04:38.355ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
37112025-05-08T22:04:38.355ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
37122025-05-08T22:04:38.355ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
37132025-05-08T22:04:38.356ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
37142025-05-08T22:04:38.356ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
37152025-05-08T22:04:38.358ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 75.167184ms iteration = 13 status = {"errors":[],"finish_invoked_ok":[]}
37162025-05-08T22:04:38.360ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
37172025-05-08T22:04:38.360ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
37182025-05-08T22:04:38.360ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
37192025-05-08T22:04:38.361ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
37202025-05-08T22:04:38.362ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37212025-05-08T22:04:38.366ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
37222025-05-08T22:04:38.366ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
37232025-05-08T22:04:38.366ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
37242025-05-08T22:04:38.367ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
37252025-05-08T22:04:38.367ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 84.20883ms iteration = 7 status = {"errors":[],"found_port_ok":[]}
37262025-05-08T22:04:38.368ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
37272025-05-08T22:04:38.369ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37282025-05-08T22:04:38.372ZDEBGnexus (BackgroundTasks): activation complete background_task = webhook_deliverator elapsed = 86.644817ms iteration = 7 status = {"by_rx":{},"error":null}
37292025-05-08T22:04:38.373ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
37302025-05-08T22:04:38.373ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
37312025-05-08T22:04:38.373ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
37322025-05-08T22:04:38.375ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37332025-05-08T22:04:38.378ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
37342025-05-08T22:04:38.378ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
37352025-05-08T22:04:38.379ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
37362025-05-08T22:04:38.380ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37372025-05-08T22:04:38.381ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37382025-05-08T22:04:38.382ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37392025-05-08T22:04:38.383ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37402025-05-08T22:04:38.387ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
37412025-05-08T22:04:38.387ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
37422025-05-08T22:04:38.387ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
37432025-05-08T22:04:38.387ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
37442025-05-08T22:04:38.387ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
37452025-05-08T22:04:38.387ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
37462025-05-08T22:04:38.388ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
37472025-05-08T22:04:38.388ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
37482025-05-08T22:04:38.388ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
37492025-05-08T22:04:38.388ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
37502025-05-08T22:04:38.388ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
37512025-05-08T22:04:38.388ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
37522025-05-08T22:04:38.389ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37532025-05-08T22:04:38.391ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37542025-05-08T22:04:38.392ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37552025-05-08T22:04:38.392ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
37562025-05-08T22:04:38.392ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
37572025-05-08T22:04:38.393ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
37582025-05-08T22:04:38.393ZDEBGnexus (BackgroundTasks): roles background_task = webhook_dispatcher roles = RoleSet { roles: {} }
37592025-05-08T22:04:38.394ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database result = Ok(())
37602025-05-08T22:04:38.396ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
37612025-05-08T22:04:38.396ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
37622025-05-08T22:04:38.396ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
37632025-05-08T22:04:38.396ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
37642025-05-08T22:04:38.396ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
37652025-05-08T22:04:38.396ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37662025-05-08T22:04:38.396ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
37672025-05-08T22:04:38.396ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
37682025-05-08T22:04:38.396ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
37692025-05-08T22:04:38.396ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
37702025-05-08T22:04:38.397ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
37712025-05-08T22:04:38.397ZDEBGnexus (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 127.119539ms iteration = 4 status = {"nsuccess":5}
37722025-05-08T22:04:38.398ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
37732025-05-08T22:04:38.398ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
37742025-05-08T22:04:38.398ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
37752025-05-08T22:04:38.398ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
37762025-05-08T22:04:38.399ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
37772025-05-08T22:04:38.401ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
37782025-05-08T22:04:38.402ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
37792025-05-08T22:04:38.408ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37802025-05-08T22:04:38.408ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
37812025-05-08T22:04:38.408ZDEBGnexus (BackgroundTasks): activation complete background_task = webhook_dispatcher elapsed = 124.798945ms iteration = 7 status = {"dispatched":[],"errors":[],"glob_version":"140.0.0","globs_reprocessed":{},"no_receivers":[]}
37822025-05-08T22:04:38.408ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
37832025-05-08T22:04:38.410ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
37842025-05-08T22:04:38.411ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
37852025-05-08T22:04:38.411ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
37862025-05-08T22:04:38.411ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
37872025-05-08T22:04:38.411ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
37882025-05-08T22:04:38.412ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
37892025-05-08T22:04:38.412ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
37902025-05-08T22:04:38.414ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 130.89588ms iteration = 13 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
37912025-05-08T22:04:38.414ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 131.088112ms iteration = 13 status = {"errors":[],"garbage_collect_requested":[]}
37922025-05-08T22:04:38.418ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1254, tv_nsec: 158958429 } })
37932025-05-08T22:04:38.418ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
37942025-05-08T22:04:38.420ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
37952025-05-08T22:04:38.420ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
37962025-05-08T22:04:38.437ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 156.090489ms iteration = 37 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
37972025-05-08T22:04:38.444ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 160.627398ms iteration = 13 status = {"errors":[],"requests_created_ok":[]}
37982025-05-08T22:04:38.449ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
37992025-05-08T22:04:38.449ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 182.007686ms iteration = 37 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
38002025-05-08T22:04:38.455ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38012025-05-08T22:04:38.456ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
38022025-05-08T22:04:38.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
38032025-05-08T22:04:38.460ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
38042025-05-08T22:04:38.460ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
38052025-05-08T22:04:38.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
38062025-05-08T22:04:38.462ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
38072025-05-08T22:04:38.462ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 7586992a-6fc5-4073-8962-728778f16aa5 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
38082025-05-08T22:04:38.462ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
38092025-05-08T22:04:38.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
38102025-05-08T22:04:38.465ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
38112025-05-08T22:04:38.465ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
38122025-05-08T22:04:38.468ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
38132025-05-08T22:04:38.468ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:137 found = 0 not_ready_to_be_deleted = 0
38142025-05-08T22:04:38.468ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 187.473595ms iteration = 7 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
38152025-05-08T22:04:38.473ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
38162025-05-08T22:04:38.473ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
38172025-05-08T22:04:38.473ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c90c012-6997-4e6d-ab83-6a133ca3dee6", "content-length": "26", "date": "Thu, 08 May 2025 22:04:38 GMT"} })
38182025-05-08T22:04:38.473ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
38192025-05-08T22:04:38.475ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
38202025-05-08T22:04:38.475ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
38212025-05-08T22:04:38.475ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = d8128696-db5a-4b88-914a-69b28494e50e dns_zone_name = oxide.test found_so_far = 0 version = 2
38222025-05-08T22:04:38.475ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
38232025-05-08T22:04:38.476ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
38242025-05-08T22:04:38.476ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
38252025-05-08T22:04:38.476ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
38262025-05-08T22:04:38.476ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 195.032705ms iteration = 13 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
38272025-05-08T22:04:38.477ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
38282025-05-08T22:04:38.477ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
38292025-05-08T22:04:38.477ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb167969-9af5-4b0a-a5b1-14a237ec6dc4", "content-length": "26", "date": "Thu, 08 May 2025 22:04:38 GMT"} })
38302025-05-08T22:04:38.477ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
38312025-05-08T22:04:38.479ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
38322025-05-08T22:04:38.481ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
38332025-05-08T22:04:38.484ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 203.034062ms iteration = 13 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
38342025-05-08T22:04:38.486ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:552 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
38352025-05-08T22:04:38.486ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 204.920542ms iteration = 13 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
38362025-05-08T22:04:38.488ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 204.648929ms iteration = 13 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
38372025-05-08T22:04:38.492ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38382025-05-08T22:04:38.495ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
38392025-05-08T22:04:38.495ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
38402025-05-08T22:04:38.495ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
38412025-05-08T22:04:38.507ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
38422025-05-08T22:04:38.508ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
38432025-05-08T22:04:38.528ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
38442025-05-08T22:04:38.529ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
38452025-05-08T22:04:38.593ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
38462025-05-08T22:04:38.631ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
38472025-05-08T22:04:38.631ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
38482025-05-08T22:04:38.641ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e zone_address = fd00:1122:3344:101::2
38492025-05-08T22:04:38.641ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
38502025-05-08T22:04:38.641ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f109b2e-14e4-4b96-81a1-651b39aa5bdc", "content-length": "26", "date": "Thu, 08 May 2025 22:04:38 GMT"} })
38512025-05-08T22:04:38.641ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1174 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
38522025-05-08T22:04:38.644ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
38532025-05-08T22:04:38.645ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
38542025-05-08T22:04:38.645ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
38552025-05-08T22:04:38.645ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6f5e915-b974-44ab-bec7-b0140d92d7ba", "content-length": "372", "date": "Thu, 08 May 2025 22:04:38 GMT"} })
38562025-05-08T22:04:38.645ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
38572025-05-08T22:04:38.646ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
38582025-05-08T22:04:38.648ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38592025-05-08T22:04:38.649ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
38602025-05-08T22:04:38.649ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
38612025-05-08T22:04:38.652ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
38622025-05-08T22:04:38.652ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
38632025-05-08T22:04:38.652ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
38642025-05-08T22:04:38.653ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
38652025-05-08T22:04:38.653ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
38662025-05-08T22:04:38.659ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 392.477287ms iteration = 13 status = null
38672025-05-08T22:04:38.698ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
38682025-05-08T22:04:38.706ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38692025-05-08T22:04:38.706ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38702025-05-08T22:04:38.707ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38712025-05-08T22:04:38.710ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
38722025-05-08T22:04:38.710ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
38732025-05-08T22:04:38.710ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
38742025-05-08T22:04:38.711ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
38752025-05-08T22:04:38.711ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
38762025-05-08T22:04:38.711ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
38772025-05-08T22:04:38.712ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 2f2f2c32-8d58-4801-a669-9618eac0c3f7, lookup_type: ById(2f2f2c32-8d58-4801-a669-9618eac0c3f7) } result = Ok(())
38782025-05-08T22:04:38.712ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
38792025-05-08T22:04:38.712ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
38802025-05-08T22:04:38.713ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = d8128696-db5a-4b88-914a-69b28494e50e dns_zone_name = oxide.test found_so_far = 1 version = 2
38812025-05-08T22:04:38.713ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
38822025-05-08T22:04:38.713ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-05-08 21:59:21.759368 UTC dns_group = external generation = 2
38832025-05-08T22:04:38.713ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 446.704468ms iteration = 7 status = {"generation":2}
38842025-05-08T22:04:38.715ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 433.768319ms iteration = 13 status = 0
38852025-05-08T22:04:38.715ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 7586992a-6fc5-4073-8962-728778f16aa5 dns_zone_name = control-plane.oxide.internal found_so_far = 55 version = 1
38862025-05-08T22:04:38.715ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
38872025-05-08T22:04:38.715ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-05-08 21:59:21.182334 UTC dns_group = internal generation = 1
38882025-05-08T22:04:38.715ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 448.259435ms iteration = 7 status = {"generation":1}
38892025-05-08T22:04:38.718ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:362
38902025-05-08T22:04:38.718ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 448.043432ms iteration = 13 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
38912025-05-08T22:04:38.726ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38922025-05-08T22:04:38.727ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
38932025-05-08T22:04:38.727ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
38942025-05-08T22:04:38.727ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "a3f13c80-353d-42c5-b937-4b6b4836d4c5", "date": "Thu, 08 May 2025 22:04:38 GMT"} })
38952025-05-08T22:04:38.727ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
38962025-05-08T22:04:38.728ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
38972025-05-08T22:04:38.728ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
38982025-05-08T22:04:38.728ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
38992025-05-08T22:04:38.728ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
39002025-05-08T22:04:38.730ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
39012025-05-08T22:04:38.730ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
39022025-05-08T22:04:38.733ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
39032025-05-08T22:04:38.733ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
39042025-05-08T22:04:38.737ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
39052025-05-08T22:04:38.737ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
39062025-05-08T22:04:38.737ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "d8b28773-e561-4177-b0d0-43e768041c86", "date": "Thu, 08 May 2025 22:04:38 GMT"} })
39072025-05-08T22:04:38.739ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 457.489063ms iteration = 13 status = {}
39082025-05-08T22:04:38.745ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
39092025-05-08T22:04:38.746ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "e933c157-3d26-4069-884c-0ac67019ad91", "date": "Thu, 08 May 2025 22:04:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
39102025-05-08T22:04:38.746ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
39112025-05-08T22:04:38.747ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "1c1ed8eb-a472-4ff5-8a80-57d8dea3402c", "date": "Thu, 08 May 2025 22:04:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
39122025-05-08T22:04:38.747ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
39132025-05-08T22:04:38.749ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
39142025-05-08T22:04:38.749ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 482.693752ms iteration = 9 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
39152025-05-08T22:04:38.767ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-05-08T21:58:35.560911Z, time_modified: 2025-05-08T21:58:35.560911Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, db_silo: Silo { identity: SiloIdentity { id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-05-08T21:59:21.709488Z, time_modified: 2025-05-08T21:59:21.709488Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, db_silo: Silo { identity: SiloIdentity { id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-05-08T21:59:21.709488Z, time_modified: 2025-05-08T21:59:21.709488Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47" }] }) }
39162025-05-08T22:04:38.767ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 500.123929ms iteration = 7 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"7137ee58-6aa7-4199-a248-feadf3b8bafb","tls_certs":["37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47"]}},"default_endpoint":{"silo_id":"7137ee58-6aa7-4199-a248-feadf3b8bafb","tls_certs":["37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
39172025-05-08T22:04:38.767ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 500.693195ms iteration = 13 status = {}
39182025-05-08T22:04:38.787ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
39192025-05-08T22:04:38.787ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
39202025-05-08T22:04:38.789ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39212025-05-08T22:04:38.791ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
39222025-05-08T22:04:38.791ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
39232025-05-08T22:04:38.791ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
39242025-05-08T22:04:38.793ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
39252025-05-08T22:04:38.794ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "75b87731-fac2-4633-80e4-190997bf03dd", "date": "Thu, 08 May 2025 22:04:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
39262025-05-08T22:04:38.794ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
39272025-05-08T22:04:38.796ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "1856399d-e705-4d5f-9dfc-ce8c6fb7c824", "date": "Thu, 08 May 2025 22:04:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
39282025-05-08T22:04:38.796ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
39292025-05-08T22:04:38.799ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "c7a7efe8-0e22-4da6-bc25-e76b42e336a7", "date": "Thu, 08 May 2025 22:04:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
39302025-05-08T22:04:38.799ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
39312025-05-08T22:04:38.801ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
39322025-05-08T22:04:38.801ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 534.76709ms iteration = 9 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
39332025-05-08T22:04:38.814ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 532.89717ms iteration = 14 status = {"physical_disks_added":0}
39342025-05-08T22:04:38.821ZINFOnexus (BackgroundTasks): instance reincarnation completed successfully background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:113 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0
39352025-05-08T22:04:38.821ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 540.014465ms iteration = 7 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
39362025-05-08T22:04:38.834ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39372025-05-08T22:04:38.836ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
39382025-05-08T22:04:38.836ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
39392025-05-08T22:04:38.836ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
39402025-05-08T22:04:38.866ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
39412025-05-08T22:04:38.866ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
39422025-05-08T22:04:38.869ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: cd0b930c-534b-46ad-ac3b-2eaa94b0bf2c, rack_id: 64f3d6f4-89b8-48aa-a622-b684e261667e, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(65f83a36-18fb-49b7-9914-87c8517d200e) } rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
39432025-05-08T22:04:38.869ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
39442025-05-08T22:04:38.870ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
39452025-05-08T22:04:38.881ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1833 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
39462025-05-08T22:04:38.881ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1860 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 65f83a36-18fb-49b7-9914-87c8517d200e
39472025-05-08T22:04:38.882ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
39482025-05-08T22:04:38.882ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
39492025-05-08T22:04:38.882ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
39502025-05-08T22:04:38.882ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
39512025-05-08T22:04:38.882ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
39522025-05-08T22:04:38.882ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2007 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
39532025-05-08T22:04:38.882ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
39542025-05-08T22:04:38.882ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2043 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
39552025-05-08T22:04:38.882ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
39562025-05-08T22:04:38.882ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
39572025-05-08T22:04:38.882ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
39582025-05-08T22:04:38.886ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
39592025-05-08T22:04:38.886ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
39602025-05-08T22:04:38.959ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
39612025-05-08T22:04:38.962ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ddbe0eb-5cdb-4957-8c9e-4769ae18c044", "content-length": "691", "date": "Thu, 08 May 2025 22:04:38 GMT"} })
39622025-05-08T22:04:38.962ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
39632025-05-08T22:04:38.962ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
39642025-05-08T22:04:38.962ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
39652025-05-08T22:04:38.963ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 682.334508ms iteration = 13 status = {}
39662025-05-08T22:04:39.011ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "649704c3-4f47-469c-8323-73a02a803b7e", "date": "Thu, 08 May 2025 22:04:38 GMT"} })
39672025-05-08T22:04:39.011ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch = Switch0
39682025-05-08T22:04:39.011ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
39692025-05-08T22:04:39.015ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "63b4c595-94bc-4d02-b1a7-dc9ed49aae95", "content-length": "682", "date": "Thu, 08 May 2025 22:04:38 GMT"} })
39702025-05-08T22:04:39.015ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
39712025-05-08T22:04:39.016ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
39722025-05-08T22:04:39.020ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
39732025-05-08T22:04:39.020ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
39742025-05-08T22:04:39.117ZDEBGnexus (producer-registration-task): registering as oximeter producer with Nexus registration_count = 2
39752025-05-08T22:04:39.117ZDEBGnexus (producer-registration-task): will register with Nexus at [fd00:1122:3344:101::b]:12221
39762025-05-08T22:04:39.136ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
39772025-05-08T22:04:39.137ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
39782025-05-08T22:04:39.139ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
39792025-05-08T22:04:39.139ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
39802025-05-08T22:04:39.145ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 863.721168ms iteration = 13 status = {}
39812025-05-08T22:04:39.149ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::b]:12221/metrics/producers
39822025-05-08T22:04:39.149ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::b]:62217
39832025-05-08T22:04:39.149ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:62217 req_id = 124d7148-0d9c-4c2a-a245-4cd4f506a971 roles = RoleSet { roles: {} } uri = /metrics/producers
39842025-05-08T22:04:39.149ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:62217 req_id = 124d7148-0d9c-4c2a-a245-4cd4f506a971 resource = Database result = Ok(()) uri = /metrics/producers
39852025-05-08T22:04:39.188ZINFOnexus: assigned collector to new producer collector_id = 22ae79e2-3731-44c9-8346-78c5ddbfe577 file = nexus/src/app/oximeter.rs:76 producer_id = 85a39cf9-9419-458f-af2e-c8b40d8f5f79
39862025-05-08T22:04:39.188ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 38658 local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:62217 req_id = 124d7148-0d9c-4c2a-a245-4cd4f506a971 response_code = 201 uri = /metrics/producers
39872025-05-08T22:04:39.188ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "124d7148-0d9c-4c2a-a245-4cd4f506a971", "content-length": "41", "date": "Thu, 08 May 2025 22:04:39 GMT"} })
39882025-05-08T22:04:39.189ZDEBGnexus (producer-registration-task): successfully registered with Nexus, pausing until time to renew lease lease_duration = 600s wait_period = 150s
39892025-05-08T22:04:41.936ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 246 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 9aa388ae-f4f4-4b4e-b1ea-26a4bdfb8e34 response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
39902025-05-08T22:04:48.268ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 38 reason = Timeout
39912025-05-08T22:04:48.269ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
39922025-05-08T22:04:48.269ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
39932025-05-08T22:04:48.270ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39942025-05-08T22:04:48.273ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
39952025-05-08T22:04:48.273ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
39962025-05-08T22:04:48.273ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
39972025-05-08T22:04:48.275ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
39982025-05-08T22:04:48.275ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
39992025-05-08T22:04:48.277ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40002025-05-08T22:04:48.280ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
40012025-05-08T22:04:48.280ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
40022025-05-08T22:04:48.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
40032025-05-08T22:04:48.281ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 38 reason = Timeout
40042025-05-08T22:04:48.281ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
40052025-05-08T22:04:48.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
40062025-05-08T22:04:48.292ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
40072025-05-08T22:04:48.292ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 23.208248ms iteration = 38 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
40082025-05-08T22:04:48.312ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
40092025-05-08T22:04:48.312ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
40102025-05-08T22:04:48.343ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 62.206945ms iteration = 38 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
40112025-05-08T22:04:51.936ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 281 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = cb2a7282-af6b-439a-a2b7-f68b3fbaf675 response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
40122025-05-08T22:04:58.268ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 39 reason = Timeout
40132025-05-08T22:04:58.269ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
40142025-05-08T22:04:58.269ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
40152025-05-08T22:04:58.271ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40162025-05-08T22:04:58.274ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
40172025-05-08T22:04:58.274ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
40182025-05-08T22:04:58.274ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
40192025-05-08T22:04:58.276ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
40202025-05-08T22:04:58.276ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
40212025-05-08T22:04:58.277ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40222025-05-08T22:04:58.280ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
40232025-05-08T22:04:58.280ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
40242025-05-08T22:04:58.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
40252025-05-08T22:04:58.281ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 39 reason = Timeout
40262025-05-08T22:04:58.281ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
40272025-05-08T22:04:58.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
40282025-05-08T22:04:58.283ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
40292025-05-08T22:04:58.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
40302025-05-08T22:04:58.291ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
40312025-05-08T22:04:58.291ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.575592ms iteration = 39 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
40322025-05-08T22:04:58.315ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 33.553538ms iteration = 39 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
40332025-05-08T22:05:01.936ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 310 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = f34b7504-056c-4a8b-81b5-b9f5d92f92d6 response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
40342025-05-08T22:05:08.266ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 14 reason = Timeout
40352025-05-08T22:05:08.266ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
40362025-05-08T22:05:08.266ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 14 reason = Timeout
40372025-05-08T22:05:08.266ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
40382025-05-08T22:05:08.266ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
40392025-05-08T22:05:08.266ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
40402025-05-08T22:05:08.268ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 40 reason = Timeout
40412025-05-08T22:05:08.268ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 14 reason = Timeout
40422025-05-08T22:05:08.268ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
40432025-05-08T22:05:08.268ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
40442025-05-08T22:05:08.269ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1284, tv_nsec: 9553301 } })
40452025-05-08T22:05:08.269ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1163
40462025-05-08T22:05:08.269ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1284, tv_nsec: 10250008 } })
40472025-05-08T22:05:08.270ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163
40482025-05-08T22:05:08.270ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 14 reason = Timeout
40492025-05-08T22:05:08.270ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40502025-05-08T22:05:08.270ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
40512025-05-08T22:05:08.270ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
40522025-05-08T22:05:08.271ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.235265ms iteration = 14 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
40532025-05-08T22:05:08.272ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40542025-05-08T22:05:08.273ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
40552025-05-08T22:05:08.273ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
40562025-05-08T22:05:08.273ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
40572025-05-08T22:05:08.274ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
40582025-05-08T22:05:08.274ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
40592025-05-08T22:05:08.275ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
40602025-05-08T22:05:08.275ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
40612025-05-08T22:05:08.275ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
40622025-05-08T22:05:08.276ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40632025-05-08T22:05:08.280ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
40642025-05-08T22:05:08.280ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
40652025-05-08T22:05:08.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
40662025-05-08T22:05:08.281ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 40 reason = Timeout
40672025-05-08T22:05:08.282ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 14 reason = Timeout
40682025-05-08T22:05:08.282ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
40692025-05-08T22:05:08.282ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 14 reason = Timeout
40702025-05-08T22:05:08.282ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
40712025-05-08T22:05:08.282ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
40722025-05-08T22:05:08.282ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
40732025-05-08T22:05:08.282ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 14 reason = Timeout
40742025-05-08T22:05:08.282ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
40752025-05-08T22:05:08.282ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 14 reason = Timeout
40762025-05-08T22:05:08.282ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
40772025-05-08T22:05:08.282ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 14 reason = Timeout
40782025-05-08T22:05:08.282ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 14 reason = Timeout
40792025-05-08T22:05:08.282ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
40802025-05-08T22:05:08.282ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
40812025-05-08T22:05:08.282ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
40822025-05-08T22:05:08.282ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 15 reason = Timeout
40832025-05-08T22:05:08.282ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
40842025-05-08T22:05:08.282ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 14 reason = Timeout
40852025-05-08T22:05:08.282ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
40862025-05-08T22:05:08.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
40872025-05-08T22:05:08.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
40882025-05-08T22:05:08.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
40892025-05-08T22:05:08.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
40902025-05-08T22:05:08.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
40912025-05-08T22:05:08.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
40922025-05-08T22:05:08.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
40932025-05-08T22:05:08.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
40942025-05-08T22:05:08.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
40952025-05-08T22:05:08.284ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 14 reason = Timeout
40962025-05-08T22:05:08.284ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
40972025-05-08T22:05:08.284ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 14 reason = Timeout
40982025-05-08T22:05:08.284ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
40992025-05-08T22:05:08.284ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 14 reason = Timeout
41002025-05-08T22:05:08.284ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
41012025-05-08T22:05:08.284ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 14 reason = Timeout
41022025-05-08T22:05:08.284ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
41032025-05-08T22:05:08.284ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 14 reason = Timeout
41042025-05-08T22:05:08.284ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
41052025-05-08T22:05:08.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
41062025-05-08T22:05:08.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
41072025-05-08T22:05:08.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
41082025-05-08T22:05:08.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
41092025-05-08T22:05:08.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
41102025-05-08T22:05:08.284ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41112025-05-08T22:05:08.285ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41122025-05-08T22:05:08.286ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
41132025-05-08T22:05:08.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
41142025-05-08T22:05:08.288ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
41152025-05-08T22:05:08.288ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
41162025-05-08T22:05:08.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
41172025-05-08T22:05:08.289ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
41182025-05-08T22:05:08.289ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
41192025-05-08T22:05:08.289ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
41202025-05-08T22:05:08.292ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41212025-05-08T22:05:08.292ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41222025-05-08T22:05:08.294ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
41232025-05-08T22:05:08.294ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 26.031429ms iteration = 40 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
41242025-05-08T22:05:08.296ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
41252025-05-08T22:05:08.296ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
41262025-05-08T22:05:08.296ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
41272025-05-08T22:05:08.296ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
41282025-05-08T22:05:08.296ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
41292025-05-08T22:05:08.296ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
41302025-05-08T22:05:08.306ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 24.113768ms iteration = 40 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
41312025-05-08T22:05:08.310ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41322025-05-08T22:05:08.310ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
41332025-05-08T22:05:08.311ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
41342025-05-08T22:05:08.312ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
41352025-05-08T22:05:08.313ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
41362025-05-08T22:05:08.313ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
41372025-05-08T22:05:08.313ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
41382025-05-08T22:05:08.313ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
41392025-05-08T22:05:08.315ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41402025-05-08T22:05:08.318ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
41412025-05-08T22:05:08.318ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
41422025-05-08T22:05:08.318ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
41432025-05-08T22:05:08.338ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
41442025-05-08T22:05:08.338ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
41452025-05-08T22:05:08.345ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
41462025-05-08T22:05:08.345ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
41472025-05-08T22:05:08.346ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "e24347db-0062-4a8d-a340-6590501d4918", "content-length": "26", "date": "Thu, 08 May 2025 22:05:08 GMT"} })
41482025-05-08T22:05:08.346ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
41492025-05-08T22:05:08.348ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
41502025-05-08T22:05:08.348ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
41512025-05-08T22:05:08.348ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
41522025-05-08T22:05:08.349ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "e65910b9-6807-4877-b15b-9b4ba2c7891e", "content-length": "26", "date": "Thu, 08 May 2025 22:05:08 GMT"} })
41532025-05-08T22:05:08.349ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
41542025-05-08T22:05:08.350ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
41552025-05-08T22:05:08.350ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
41562025-05-08T22:05:08.352ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
41572025-05-08T22:05:08.352ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
41582025-05-08T22:05:08.353ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
41592025-05-08T22:05:08.353ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
41602025-05-08T22:05:08.353ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
41612025-05-08T22:05:08.353ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
41622025-05-08T22:05:08.354ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 70.114259ms iteration = 14 status = {"errors":[],"requests_created_ok":[]}
41632025-05-08T22:05:08.354ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
41642025-05-08T22:05:08.354ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41652025-05-08T22:05:08.357ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
41662025-05-08T22:05:08.357ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
41672025-05-08T22:05:08.363ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 2f2f2c32-8d58-4801-a669-9618eac0c3f7, lookup_type: ById(2f2f2c32-8d58-4801-a669-9618eac0c3f7) } result = Ok(())
41682025-05-08T22:05:08.363ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
41692025-05-08T22:05:08.363ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
41702025-05-08T22:05:08.364ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
41712025-05-08T22:05:08.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
41722025-05-08T22:05:08.365ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 81.470691ms iteration = 14 status = {"errors":[],"garbage_collect_requested":[]}
41732025-05-08T22:05:08.366ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 84.099279ms iteration = 14 status = 0
41742025-05-08T22:05:08.366ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41752025-05-08T22:05:08.368ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:552 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
41762025-05-08T22:05:08.368ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 85.946369ms iteration = 14 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
41772025-05-08T22:05:08.368ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
41782025-05-08T22:05:08.368ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
41792025-05-08T22:05:08.370ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
41802025-05-08T22:05:08.370ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
41812025-05-08T22:05:08.370ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
41822025-05-08T22:05:08.371ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 87.152842ms iteration = 14 status = {"errors":[],"finish_invoked_ok":[]}
41832025-05-08T22:05:08.373ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1284, tv_nsec: 114134249 } })
41842025-05-08T22:05:08.373ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
41852025-05-08T22:05:08.390ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 108.793914ms iteration = 15 status = {"physical_disks_added":0}
41862025-05-08T22:05:08.396ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 114.202541ms iteration = 14 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
41872025-05-08T22:05:08.397ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41882025-05-08T22:05:08.399ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
41892025-05-08T22:05:08.399ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
41902025-05-08T22:05:08.399ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
41912025-05-08T22:05:08.411ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 126.951957ms iteration = 14 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
41922025-05-08T22:05:08.412ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
41932025-05-08T22:05:08.412ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
41942025-05-08T22:05:08.412ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
41952025-05-08T22:05:08.412ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
41962025-05-08T22:05:08.422ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
41972025-05-08T22:05:08.423ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
41982025-05-08T22:05:08.443ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
41992025-05-08T22:05:08.444ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 177.716741ms iteration = 14 status = {}
42002025-05-08T22:05:08.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
42012025-05-08T22:05:08.450ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:362
42022025-05-08T22:05:08.450ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 180.087586ms iteration = 14 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
42032025-05-08T22:05:08.450ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
42042025-05-08T22:05:08.450ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
42052025-05-08T22:05:08.450ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 168.747645ms iteration = 14 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
42062025-05-08T22:05:08.468ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
42072025-05-08T22:05:08.468ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
42082025-05-08T22:05:08.516ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e zone_address = fd00:1122:3344:101::2
42092025-05-08T22:05:08.516ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
42102025-05-08T22:05:08.516ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c58bced-47e8-44ee-b0c2-98d27e6591d0", "content-length": "26", "date": "Thu, 08 May 2025 22:05:08 GMT"} })
42112025-05-08T22:05:08.516ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1174 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
42122025-05-08T22:05:08.519ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
42132025-05-08T22:05:08.536ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
42142025-05-08T22:05:08.536ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
42152025-05-08T22:05:08.536ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "ccdde56c-a32b-427a-9777-336d398023c7", "content-length": "372", "date": "Thu, 08 May 2025 22:05:08 GMT"} })
42162025-05-08T22:05:08.536ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
42172025-05-08T22:05:08.537ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
42182025-05-08T22:05:08.538ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 254.333901ms iteration = 14 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
42192025-05-08T22:05:08.539ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 273.411344ms iteration = 14 status = null
42202025-05-08T22:05:08.600ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
42212025-05-08T22:05:08.600ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
42222025-05-08T22:05:08.600ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "a8d9ecb6-1b3a-45c5-aaaa-7c1994ac0b3f", "date": "Thu, 08 May 2025 22:05:08 GMT"} })
42232025-05-08T22:05:08.600ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
42242025-05-08T22:05:08.601ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
42252025-05-08T22:05:08.603ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
42262025-05-08T22:05:08.603ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
42272025-05-08T22:05:08.605ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
42282025-05-08T22:05:08.605ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
42292025-05-08T22:05:08.609ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
42302025-05-08T22:05:08.609ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
42312025-05-08T22:05:08.634ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
42322025-05-08T22:05:08.634ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
42332025-05-08T22:05:08.636ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: cd0b930c-534b-46ad-ac3b-2eaa94b0bf2c, rack_id: 64f3d6f4-89b8-48aa-a622-b684e261667e, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(65f83a36-18fb-49b7-9914-87c8517d200e) } rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
42342025-05-08T22:05:08.636ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
42352025-05-08T22:05:08.636ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
42362025-05-08T22:05:08.648ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1833 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
42372025-05-08T22:05:08.648ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1860 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 65f83a36-18fb-49b7-9914-87c8517d200e
42382025-05-08T22:05:08.648ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
42392025-05-08T22:05:08.648ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
42402025-05-08T22:05:08.648ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
42412025-05-08T22:05:08.648ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
42422025-05-08T22:05:08.648ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
42432025-05-08T22:05:08.648ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2007 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
42442025-05-08T22:05:08.648ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
42452025-05-08T22:05:08.648ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2043 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
42462025-05-08T22:05:08.648ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
42472025-05-08T22:05:08.648ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
42482025-05-08T22:05:08.649ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
42492025-05-08T22:05:08.649ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
42502025-05-08T22:05:08.649ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
42512025-05-08T22:05:08.657ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42522025-05-08T22:05:08.660ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
42532025-05-08T22:05:08.660ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
42542025-05-08T22:05:08.660ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
42552025-05-08T22:05:08.660ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
42562025-05-08T22:05:08.660ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
42572025-05-08T22:05:08.661ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42582025-05-08T22:05:08.663ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
42592025-05-08T22:05:08.663ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
42602025-05-08T22:05:08.663ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
42612025-05-08T22:05:08.701ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
42622025-05-08T22:05:08.727ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "186e892d-5370-43aa-92fa-6e243733fc96", "content-length": "691", "date": "Thu, 08 May 2025 22:05:08 GMT"} })
42632025-05-08T22:05:08.727ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
42642025-05-08T22:05:08.727ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
42652025-05-08T22:05:08.727ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
42662025-05-08T22:05:08.728ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 446.479286ms iteration = 14 status = {}
42672025-05-08T22:05:08.732ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "c37c7237-3202-4017-85f4-6ece4da0f609", "date": "Thu, 08 May 2025 22:05:08 GMT"} })
42682025-05-08T22:05:08.735ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 452.993485ms iteration = 14 status = {}
42692025-05-08T22:05:08.827ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "9eae982d-805b-4812-815d-dba6592e2271", "date": "Thu, 08 May 2025 22:05:08 GMT"} })
42702025-05-08T22:05:08.827ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch = Switch0
42712025-05-08T22:05:08.827ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
42722025-05-08T22:05:08.828ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "406faa3f-c9a4-40a5-9b9f-474c9ce6f095", "content-length": "682", "date": "Thu, 08 May 2025 22:05:08 GMT"} })
42732025-05-08T22:05:08.828ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
42742025-05-08T22:05:08.828ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
42752025-05-08T22:05:08.832ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
42762025-05-08T22:05:08.832ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
42772025-05-08T22:05:08.835ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
42782025-05-08T22:05:08.836ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
42792025-05-08T22:05:08.838ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
42802025-05-08T22:05:08.838ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
42812025-05-08T22:05:08.845ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 563.351245ms iteration = 14 status = {}
42822025-05-08T22:05:11.937ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 500 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = a0ad6be6-8722-4a62-8f3a-26369040f867 response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
42832025-05-08T22:05:18.268ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 41 reason = Timeout
42842025-05-08T22:05:18.268ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
42852025-05-08T22:05:18.268ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
42862025-05-08T22:05:18.270ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42872025-05-08T22:05:18.273ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
42882025-05-08T22:05:18.273ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
42892025-05-08T22:05:18.273ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
42902025-05-08T22:05:18.274ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
42912025-05-08T22:05:18.275ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
42922025-05-08T22:05:18.276ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42932025-05-08T22:05:18.279ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
42942025-05-08T22:05:18.279ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
42952025-05-08T22:05:18.279ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
42962025-05-08T22:05:18.281ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 41 reason = Timeout
42972025-05-08T22:05:18.281ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
42982025-05-08T22:05:18.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
42992025-05-08T22:05:18.283ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
43002025-05-08T22:05:18.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
43012025-05-08T22:05:18.291ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
43022025-05-08T22:05:18.291ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.895735ms iteration = 41 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
43032025-05-08T22:05:18.315ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 33.554489ms iteration = 41 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
43042025-05-08T22:05:21.929ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::d]:51552
43052025-05-08T22:05:21.929ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:51552 req_id = 67fa66f2-9f1e-48f2-b674-d62b4f5890e9 roles = RoleSet { roles: {} } uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?sort_by=id_ascending
43062025-05-08T22:05:21.930ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:51552 req_id = 67fa66f2-9f1e-48f2-b674-d62b4f5890e9 resource = Database result = Ok(()) uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?sort_by=id_ascending
43072025-05-08T22:05:21.932ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2953 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:51552 req_id = 67fa66f2-9f1e-48f2-b674-d62b4f5890e9 response_code = 200 uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?sort_by=id_ascending
43082025-05-08T22:05:21.932ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:51552 req_id = 5f424b97-529a-4220-a1d5-8d7ef1565815 roles = RoleSet { roles: {} } uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZGQ4MjYzNC0yODQ0LTRmOTctOTc0Ni0wNzA5OTA0NGMxOWQifX0%3D
43092025-05-08T22:05:21.933ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:51552 req_id = 5f424b97-529a-4220-a1d5-8d7ef1565815 resource = Database result = Ok(()) uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZGQ4MjYzNC0yODQ0LTRmOTctOTc0Ni0wNzA5OTA0NGMxOWQifX0%3D
43102025-05-08T22:05:21.935ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2357 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:51552 req_id = 5f424b97-529a-4220-a1d5-8d7ef1565815 response_code = 200 uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZGQ4MjYzNC0yODQ0LTRmOTctOTc0Ni0wNzA5OTA0NGMxOWQifX0%3D
43112025-05-08T22:05:21.936ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 245 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 48b54dce-4011-477e-ad18-2adfdff318fc response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
43122025-05-08T22:05:28.268ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 42 reason = Timeout
43132025-05-08T22:05:28.269ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
43142025-05-08T22:05:28.269ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
43152025-05-08T22:05:28.271ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43162025-05-08T22:05:28.274ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
43172025-05-08T22:05:28.274ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
43182025-05-08T22:05:28.274ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
43192025-05-08T22:05:28.276ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
43202025-05-08T22:05:28.276ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
43212025-05-08T22:05:28.277ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43222025-05-08T22:05:28.280ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
43232025-05-08T22:05:28.280ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
43242025-05-08T22:05:28.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
43252025-05-08T22:05:28.282ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 42 reason = Timeout
43262025-05-08T22:05:28.282ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
43272025-05-08T22:05:28.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
43282025-05-08T22:05:28.283ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
43292025-05-08T22:05:28.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
43302025-05-08T22:05:28.284ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 2.835181ms iteration = 42 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
43312025-05-08T22:05:28.291ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
43322025-05-08T22:05:28.291ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.249418ms iteration = 42 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
43332025-05-08T22:05:31.937ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 411 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 01a27605-8566-4736-9ca0-22c09f790a49 response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
43342025-05-08T22:05:38.267ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 15 reason = Timeout
43352025-05-08T22:05:38.267ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
43362025-05-08T22:05:38.267ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 10 reason = Timeout
43372025-05-08T22:05:38.267ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 8 reason = Timeout
43382025-05-08T22:05:38.267ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 8 reason = Timeout
43392025-05-08T22:05:38.267ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
43402025-05-08T22:05:38.267ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
43412025-05-08T22:05:38.267ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 8 reason = Timeout
43422025-05-08T22:05:38.267ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
43432025-05-08T22:05:38.267ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 15 reason = Timeout
43442025-05-08T22:05:38.267ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 10 reason = Timeout
43452025-05-08T22:05:38.267ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
43462025-05-08T22:05:38.267ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 8 reason = Timeout
43472025-05-08T22:05:38.267ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-05-08 21:55:38.267236012 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
43482025-05-08T22:05:38.267ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
43492025-05-08T22:05:38.267ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 8 reason = Timeout
43502025-05-08T22:05:38.267ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 8 reason = Timeout
43512025-05-08T22:05:38.267ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
43522025-05-08T22:05:38.267ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
43532025-05-08T22:05:38.267ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
43542025-05-08T22:05:38.267ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
43552025-05-08T22:05:38.267ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
43562025-05-08T22:05:38.267ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
43572025-05-08T22:05:38.267ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
43582025-05-08T22:05:38.267ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
43592025-05-08T22:05:38.267ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("228ed1c1-3cc1-4055-aae9-5595ad856b86.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("8b245480-f91d-406b-bb08-850e1c3e3fc5.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c8f20f67-ea63-4dbc-b0f1-c30876daff82.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c8f20f67-ea63-4dbc-b0f1-c30876daff82.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 1314, tv_nsec: 8217317 } })
43602025-05-08T22:05:38.267ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("0ccabf41-81cd-45e7-bb26-335edaa2ee47.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("38a08dd6-6a75-4558-b4f8-df853f4d03d9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("38a08dd6-6a75-4558-b4f8-df853f4d03d9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::9))) }], valid_until: Instant { tv_sec: 1314, tv_nsec: 8232917 } })
43612025-05-08T22:05:38.268ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 43 reason = Timeout
43622025-05-08T22:05:38.268ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } dns_group = external
43632025-05-08T22:05:38.268ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.334564ms iteration = 8 status = {"addresses":["[fd00:1122:3344:101::8]:5353","[fd00:1122:3344:101::9]:5353"]}
43642025-05-08T22:05:38.268ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 9 reason = Timeout
43652025-05-08T22:05:38.268ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:85 reason = blueprint disabled target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2
43662025-05-08T22:05:38.268ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
43672025-05-08T22:05:38.268ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 11.32\u{b5}s iteration = 9 status = {"enabled":false,"target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2"}
43682025-05-08T22:05:38.268ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 15 reason = Timeout
43692025-05-08T22:05:38.268ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
43702025-05-08T22:05:38.268ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
43712025-05-08T22:05:38.268ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.304054ms iteration = 8 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:1::1]:5353"]}
43722025-05-08T22:05:38.268ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
43732025-05-08T22:05:38.268ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
43742025-05-08T22:05:38.270ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 15 reason = Timeout
43752025-05-08T22:05:38.270ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
43762025-05-08T22:05:38.270ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
43772025-05-08T22:05:38.270ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1314, tv_nsec: 11121728 } })
43782025-05-08T22:05:38.271ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163
43792025-05-08T22:05:38.271ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
43802025-05-08T22:05:38.271ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
43812025-05-08T22:05:38.271ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1314, tv_nsec: 12108928 } })
43822025-05-08T22:05:38.272ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-05-08 21:55:38.267236012 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
43832025-05-08T22:05:38.272ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 4.846042ms iteration = 8 status = {"expiration":"2025-05-08T21:55:38.267236012Z","pruned":[]}
43842025-05-08T22:05:38.272ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1163
43852025-05-08T22:05:38.272ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43862025-05-08T22:05:38.274ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
43872025-05-08T22:05:38.274ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
43882025-05-08T22:05:38.274ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
43892025-05-08T22:05:38.275ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 6.54167ms iteration = 15 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
43902025-05-08T22:05:38.276ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43912025-05-08T22:05:38.277ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
43922025-05-08T22:05:38.277ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
43932025-05-08T22:05:38.279ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
43942025-05-08T22:05:38.281ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 43 reason = Timeout
43952025-05-08T22:05:38.281ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
43962025-05-08T22:05:38.281ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 15 reason = Timeout
43972025-05-08T22:05:38.281ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 16 reason = Timeout
43982025-05-08T22:05:38.281ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
43992025-05-08T22:05:38.281ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
44002025-05-08T22:05:38.281ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 8 reason = Timeout
44012025-05-08T22:05:38.281ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:53
44022025-05-08T22:05:38.281ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 8 reason = Timeout
44032025-05-08T22:05:38.281ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
44042025-05-08T22:05:38.281ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 15 reason = Timeout
44052025-05-08T22:05:38.281ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
44062025-05-08T22:05:38.281ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
44072025-05-08T22:05:38.281ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 15 reason = Timeout
44082025-05-08T22:05:38.281ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
44092025-05-08T22:05:38.281ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 15 reason = Timeout
44102025-05-08T22:05:38.281ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
44112025-05-08T22:05:38.281ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
44122025-05-08T22:05:38.281ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 15 reason = Timeout
44132025-05-08T22:05:38.281ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
44142025-05-08T22:05:38.281ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
44152025-05-08T22:05:38.281ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 15 reason = Timeout
44162025-05-08T22:05:38.281ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 15 reason = Timeout
44172025-05-08T22:05:38.281ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
44182025-05-08T22:05:38.282ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
44192025-05-08T22:05:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
44202025-05-08T22:05:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
44212025-05-08T22:05:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
44222025-05-08T22:05:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
44232025-05-08T22:05:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
44242025-05-08T22:05:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
44252025-05-08T22:05:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
44262025-05-08T22:05:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
44272025-05-08T22:05:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
44282025-05-08T22:05:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
44292025-05-08T22:05:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
44302025-05-08T22:05:38.283ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
44312025-05-08T22:05:38.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
44322025-05-08T22:05:38.283ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 15 reason = Timeout
44332025-05-08T22:05:38.283ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
44342025-05-08T22:05:38.284ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 15 reason = Timeout
44352025-05-08T22:05:38.284ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 15 reason = Timeout
44362025-05-08T22:05:38.284ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
44372025-05-08T22:05:38.284ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
44382025-05-08T22:05:38.284ZDEBGnexus (BackgroundTasks): activating background_task = webhook_deliverator iteration = 8 reason = Timeout
44392025-05-08T22:05:38.284ZDEBGnexus (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} }
44402025-05-08T22:05:38.284ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 8 reason = Timeout
44412025-05-08T22:05:38.284ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
44422025-05-08T22:05:38.284ZDEBGnexus (BackgroundTasks): activating background_task = webhook_dispatcher iteration = 8 reason = Timeout
44432025-05-08T22:05:38.284ZDEBGnexus (BackgroundTasks): roles background_task = webhook_dispatcher roles = RoleSet { roles: {} }
44442025-05-08T22:05:38.284ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 8 reason = Timeout
44452025-05-08T22:05:38.284ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
44462025-05-08T22:05:38.284ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 15 reason = Timeout
44472025-05-08T22:05:38.284ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
44482025-05-08T22:05:38.284ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 15 reason = Timeout
44492025-05-08T22:05:38.284ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
44502025-05-08T22:05:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
44512025-05-08T22:05:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
44522025-05-08T22:05:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
44532025-05-08T22:05:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
44542025-05-08T22:05:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database result = Ok(())
44552025-05-08T22:05:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
44562025-05-08T22:05:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
44572025-05-08T22:05:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database result = Ok(())
44582025-05-08T22:05:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
44592025-05-08T22:05:38.284ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44602025-05-08T22:05:38.284ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44612025-05-08T22:05:38.288ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
44622025-05-08T22:05:38.288ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
44632025-05-08T22:05:38.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
44642025-05-08T22:05:38.290ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
44652025-05-08T22:05:38.290ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
44662025-05-08T22:05:38.290ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
44672025-05-08T22:05:38.292ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44682025-05-08T22:05:38.294ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44692025-05-08T22:05:38.296ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
44702025-05-08T22:05:38.296ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
44712025-05-08T22:05:38.296ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
44722025-05-08T22:05:38.297ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
44732025-05-08T22:05:38.297ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
44742025-05-08T22:05:38.297ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
44752025-05-08T22:05:38.318ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 34.540029ms iteration = 8 status = {"errors":[],"found_port_ok":[]}
44762025-05-08T22:05:38.320ZDEBGnexus (BackgroundTasks): activation complete background_task = webhook_deliverator elapsed = 35.973214ms iteration = 8 status = {"by_rx":{},"error":null}
44772025-05-08T22:05:38.330ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44782025-05-08T22:05:38.332ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44792025-05-08T22:05:38.335ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
44802025-05-08T22:05:38.337ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
44812025-05-08T22:05:38.337ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
44822025-05-08T22:05:38.337ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
44832025-05-08T22:05:38.341ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44842025-05-08T22:05:38.345ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
44852025-05-08T22:05:38.345ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
44862025-05-08T22:05:38.345ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
44872025-05-08T22:05:38.345ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44882025-05-08T22:05:38.347ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
44892025-05-08T22:05:38.347ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
44902025-05-08T22:05:38.347ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
44912025-05-08T22:05:38.347ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
44922025-05-08T22:05:38.348ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
44932025-05-08T22:05:38.348ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:552 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
44942025-05-08T22:05:38.348ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 66.537442ms iteration = 15 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
44952025-05-08T22:05:38.348ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
44962025-05-08T22:05:38.349ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
44972025-05-08T22:05:38.350ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44982025-05-08T22:05:38.350ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
44992025-05-08T22:05:38.350ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
45002025-05-08T22:05:38.350ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
45012025-05-08T22:05:38.353ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
45022025-05-08T22:05:38.353ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
45032025-05-08T22:05:38.353ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
45042025-05-08T22:05:38.358ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
45052025-05-08T22:05:38.358ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
45062025-05-08T22:05:38.358ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
45072025-05-08T22:05:38.358ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
45082025-05-08T22:05:38.359ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
45092025-05-08T22:05:38.360ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45102025-05-08T22:05:38.360ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
45112025-05-08T22:05:38.362ZDEBGnexus (BackgroundTasks): roles background_task = webhook_dispatcher roles = RoleSet { roles: {} }
45122025-05-08T22:05:38.363ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database result = Ok(())
45132025-05-08T22:05:38.363ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
45142025-05-08T22:05:38.363ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
45152025-05-08T22:05:38.363ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
45162025-05-08T22:05:38.364ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45172025-05-08T22:05:38.364ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1314, tv_nsec: 104945131 } })
45182025-05-08T22:05:38.365ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
45192025-05-08T22:05:38.365ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
45202025-05-08T22:05:38.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
45212025-05-08T22:05:38.366ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 85.10471ms iteration = 43 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
45222025-05-08T22:05:38.368ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
45232025-05-08T22:05:38.368ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
45242025-05-08T22:05:38.368ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
45252025-05-08T22:05:38.374ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 92.495879ms iteration = 15 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
45262025-05-08T22:05:38.374ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
45272025-05-08T22:05:38.374ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
45282025-05-08T22:05:38.374ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
45292025-05-08T22:05:38.375ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
45302025-05-08T22:05:38.383ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45312025-05-08T22:05:38.384ZDEBGnexus (BackgroundTasks): activation complete background_task = webhook_dispatcher elapsed = 100.854008ms iteration = 8 status = {"dispatched":[],"errors":[],"glob_version":"140.0.0","globs_reprocessed":{},"no_receivers":[]}
45322025-05-08T22:05:38.384ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
45332025-05-08T22:05:38.384ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
45342025-05-08T22:05:38.384ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 100.876659ms iteration = 8 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
45352025-05-08T22:05:38.386ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
45362025-05-08T22:05:38.386ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
45372025-05-08T22:05:38.387ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 103.604028ms iteration = 15 status = {"errors":[],"finish_invoked_ok":[]}
45382025-05-08T22:05:38.388ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
45392025-05-08T22:05:38.388ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
45402025-05-08T22:05:38.388ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 103.949092ms iteration = 15 status = {"errors":[],"requests_created_ok":[]}
45412025-05-08T22:05:38.388ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
45422025-05-08T22:05:38.388ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
45432025-05-08T22:05:38.388ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
45442025-05-08T22:05:38.391ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45452025-05-08T22:05:38.395ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
45462025-05-08T22:05:38.395ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
45472025-05-08T22:05:38.396ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
45482025-05-08T22:05:38.397ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 113.910218ms iteration = 15 status = {"errors":[],"garbage_collect_requested":[]}
45492025-05-08T22:05:38.398ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
45502025-05-08T22:05:38.399ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
45512025-05-08T22:05:38.400ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 116.787799ms iteration = 15 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
45522025-05-08T22:05:38.407ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
45532025-05-08T22:05:38.407ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 139.27497ms iteration = 43 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
45542025-05-08T22:05:38.409ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
45552025-05-08T22:05:38.409ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
45562025-05-08T22:05:38.409ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
45572025-05-08T22:05:38.409ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
45582025-05-08T22:05:38.409ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
45592025-05-08T22:05:38.411ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45602025-05-08T22:05:38.416ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
45612025-05-08T22:05:38.417ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
45622025-05-08T22:05:38.417ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
45632025-05-08T22:05:38.418ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
45642025-05-08T22:05:38.419ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
45652025-05-08T22:05:38.427ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
45662025-05-08T22:05:38.428ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
45672025-05-08T22:05:38.430ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45682025-05-08T22:05:38.430ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
45692025-05-08T22:05:38.430ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45702025-05-08T22:05:38.434ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
45712025-05-08T22:05:38.434ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
45722025-05-08T22:05:38.434ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
45732025-05-08T22:05:38.437ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 2f2f2c32-8d58-4801-a669-9618eac0c3f7, lookup_type: ById(2f2f2c32-8d58-4801-a669-9618eac0c3f7) } result = Ok(())
45742025-05-08T22:05:38.437ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
45752025-05-08T22:05:38.438ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
45762025-05-08T22:05:38.442ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 161.023712ms iteration = 15 status = 0
45772025-05-08T22:05:38.448ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
45782025-05-08T22:05:38.448ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
45792025-05-08T22:05:38.448ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "8cb9b0ad-fccb-483a-aa55-993842115211", "content-length": "26", "date": "Thu, 08 May 2025 22:05:38 GMT"} })
45802025-05-08T22:05:38.448ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
45812025-05-08T22:05:38.451ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
45822025-05-08T22:05:38.453ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
45832025-05-08T22:05:38.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
45842025-05-08T22:05:38.454ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
45852025-05-08T22:05:38.454ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:137 found = 0 not_ready_to_be_deleted = 0
45862025-05-08T22:05:38.454ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 172.199912ms iteration = 8 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
45872025-05-08T22:05:38.455ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45882025-05-08T22:05:38.457ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
45892025-05-08T22:05:38.457ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
45902025-05-08T22:05:38.457ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
45912025-05-08T22:05:38.457ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
45922025-05-08T22:05:38.457ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 175.963482ms iteration = 15 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
45932025-05-08T22:05:38.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
45942025-05-08T22:05:38.458ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
45952025-05-08T22:05:38.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
45962025-05-08T22:05:38.464ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
45972025-05-08T22:05:38.465ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
45982025-05-08T22:05:38.465ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45992025-05-08T22:05:38.468ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
46002025-05-08T22:05:38.468ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
46012025-05-08T22:05:38.468ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
46022025-05-08T22:05:38.474ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
46032025-05-08T22:05:38.483ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
46042025-05-08T22:05:38.483ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
46052025-05-08T22:05:38.483ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = d8128696-db5a-4b88-914a-69b28494e50e dns_zone_name = oxide.test found_so_far = 0 version = 2
46062025-05-08T22:05:38.483ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
46072025-05-08T22:05:38.484ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
46082025-05-08T22:05:38.484ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
46092025-05-08T22:05:38.485ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46102025-05-08T22:05:38.491ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
46112025-05-08T22:05:38.491ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
46122025-05-08T22:05:38.491ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
46132025-05-08T22:05:38.493ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = d8128696-db5a-4b88-914a-69b28494e50e dns_zone_name = oxide.test found_so_far = 1 version = 2
46142025-05-08T22:05:38.493ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
46152025-05-08T22:05:38.493ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-05-08 21:59:21.759368 UTC dns_group = external generation = 2
46162025-05-08T22:05:38.493ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 226.557473ms iteration = 8 status = {"generation":2}
46172025-05-08T22:05:38.499ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
46182025-05-08T22:05:38.499ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 7586992a-6fc5-4073-8962-728778f16aa5 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
46192025-05-08T22:05:38.499ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
46202025-05-08T22:05:38.499ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
46212025-05-08T22:05:38.501ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46222025-05-08T22:05:38.509ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
46232025-05-08T22:05:38.509ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
46242025-05-08T22:05:38.509ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
46252025-05-08T22:05:38.513ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 7586992a-6fc5-4073-8962-728778f16aa5 dns_zone_name = control-plane.oxide.internal found_so_far = 55 version = 1
46262025-05-08T22:05:38.513ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
46272025-05-08T22:05:38.513ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-05-08 21:59:21.182334 UTC dns_group = internal generation = 1
46282025-05-08T22:05:38.513ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 246.348074ms iteration = 8 status = {"generation":1}
46292025-05-08T22:05:38.518ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
46302025-05-08T22:05:38.518ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 234.150614ms iteration = 15 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
46312025-05-08T22:05:38.518ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
46322025-05-08T22:05:38.525ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-05-08T21:58:35.560911Z, time_modified: 2025-05-08T21:58:35.560911Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, db_silo: Silo { identity: SiloIdentity { id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-05-08T21:59:21.709488Z, time_modified: 2025-05-08T21:59:21.709488Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, db_silo: Silo { identity: SiloIdentity { id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-05-08T21:59:21.709488Z, time_modified: 2025-05-08T21:59:21.709488Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47" }] }) }
46332025-05-08T22:05:38.525ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 254.529352ms iteration = 8 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"7137ee58-6aa7-4199-a248-feadf3b8bafb","tls_certs":["37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47"]}},"default_endpoint":{"silo_id":"7137ee58-6aa7-4199-a248-feadf3b8bafb","tls_certs":["37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
46342025-05-08T22:05:38.532ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e zone_address = fd00:1122:3344:101::2
46352025-05-08T22:05:38.532ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
46362025-05-08T22:05:38.532ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c193320-423f-4af3-a55a-c15cf0f4747a", "content-length": "26", "date": "Thu, 08 May 2025 22:05:38 GMT"} })
46372025-05-08T22:05:38.532ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1174 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
46382025-05-08T22:05:38.534ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
46392025-05-08T22:05:38.545ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:362
46402025-05-08T22:05:38.545ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 274.507656ms iteration = 15 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
46412025-05-08T22:05:38.572ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 290.487307ms iteration = 16 status = {"physical_disks_added":0}
46422025-05-08T22:05:38.606ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
46432025-05-08T22:05:38.606ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
46442025-05-08T22:05:38.606ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ad8d6ba-f915-43b5-82cc-ef19af9ce7f1", "content-length": "26", "date": "Thu, 08 May 2025 22:05:38 GMT"} })
46452025-05-08T22:05:38.606ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
46462025-05-08T22:05:38.608ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
46472025-05-08T22:05:38.634ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
46482025-05-08T22:05:38.634ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "c714ca77-7a9d-4b0a-8ae2-21297dc87aed", "date": "Thu, 08 May 2025 22:05:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
46492025-05-08T22:05:38.634ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
46502025-05-08T22:05:38.636ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "321551e4-afa6-4906-97b0-0d2f5be29ef3", "date": "Thu, 08 May 2025 22:05:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
46512025-05-08T22:05:38.636ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
46522025-05-08T22:05:38.640ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
46532025-05-08T22:05:38.640ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 373.142862ms iteration = 10 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
46542025-05-08T22:05:38.647ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
46552025-05-08T22:05:38.647ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
46562025-05-08T22:05:38.648ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "485c7e58-d139-4942-a189-ab13b51d2ff0", "content-length": "372", "date": "Thu, 08 May 2025 22:05:38 GMT"} })
46572025-05-08T22:05:38.648ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
46582025-05-08T22:05:38.648ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
46592025-05-08T22:05:38.649ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 382.37482ms iteration = 15 status = {}
46602025-05-08T22:05:38.696ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
46612025-05-08T22:05:38.696ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
46622025-05-08T22:05:38.698ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46632025-05-08T22:05:38.700ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
46642025-05-08T22:05:38.700ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
46652025-05-08T22:05:38.700ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
46662025-05-08T22:05:38.702ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46672025-05-08T22:05:38.706ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
46682025-05-08T22:05:38.706ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
46692025-05-08T22:05:38.706ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
46702025-05-08T22:05:38.707ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
46712025-05-08T22:05:38.707ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
46722025-05-08T22:05:38.707ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "85a7cce1-3e9f-4638-bf88-5e82ca22c9be", "date": "Thu, 08 May 2025 22:05:38 GMT"} })
46732025-05-08T22:05:38.707ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
46742025-05-08T22:05:38.708ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
46752025-05-08T22:05:38.710ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
46762025-05-08T22:05:38.710ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
46772025-05-08T22:05:38.711ZINFOnexus (BackgroundTasks): instance reincarnation completed successfully background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:113 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0
46782025-05-08T22:05:38.711ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 429.234151ms iteration = 8 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
46792025-05-08T22:05:38.722ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
46802025-05-08T22:05:38.722ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
46812025-05-08T22:05:38.725ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
46822025-05-08T22:05:38.725ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
46832025-05-08T22:05:38.726ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "1587a052-30ca-410f-a083-f640d07045a9", "date": "Thu, 08 May 2025 22:05:38 GMT"} })
46842025-05-08T22:05:38.727ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 446.089401ms iteration = 15 status = {}
46852025-05-08T22:05:38.778ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
46862025-05-08T22:05:38.832ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
46872025-05-08T22:05:38.832ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
46882025-05-08T22:05:38.839ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
46892025-05-08T22:05:38.840ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "9570e891-ea9f-4371-b6cc-3216182675ae", "content-length": "691", "date": "Thu, 08 May 2025 22:05:38 GMT"} })
46902025-05-08T22:05:38.840ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
46912025-05-08T22:05:38.840ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
46922025-05-08T22:05:38.840ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
46932025-05-08T22:05:38.841ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 559.966569ms iteration = 15 status = {}
46942025-05-08T22:05:38.842ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 574.91203ms iteration = 15 status = null
46952025-05-08T22:05:38.870ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
46962025-05-08T22:05:38.870ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
46972025-05-08T22:05:38.872ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: cd0b930c-534b-46ad-ac3b-2eaa94b0bf2c, rack_id: 64f3d6f4-89b8-48aa-a622-b684e261667e, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(65f83a36-18fb-49b7-9914-87c8517d200e) } rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
46982025-05-08T22:05:38.872ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
46992025-05-08T22:05:38.873ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
47002025-05-08T22:05:38.877ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
47012025-05-08T22:05:38.877ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "15f2a41f-f631-48d0-8c81-b18c74229219", "date": "Thu, 08 May 2025 22:05:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
47022025-05-08T22:05:38.877ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
47032025-05-08T22:05:38.878ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "90eddf7f-2401-4450-8bf2-b3a719f11d77", "date": "Thu, 08 May 2025 22:05:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
47042025-05-08T22:05:38.878ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
47052025-05-08T22:05:38.880ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "be37c835-d0b0-405c-b57b-6be07a50f7cc", "date": "Thu, 08 May 2025 22:05:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
47062025-05-08T22:05:38.880ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
47072025-05-08T22:05:38.882ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
47082025-05-08T22:05:38.882ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 615.377172ms iteration = 10 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
47092025-05-08T22:05:38.883ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1833 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
47102025-05-08T22:05:38.883ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1860 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 65f83a36-18fb-49b7-9914-87c8517d200e
47112025-05-08T22:05:38.883ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
47122025-05-08T22:05:38.883ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
47132025-05-08T22:05:38.883ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
47142025-05-08T22:05:38.883ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
47152025-05-08T22:05:38.883ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
47162025-05-08T22:05:38.883ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2007 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
47172025-05-08T22:05:38.883ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
47182025-05-08T22:05:38.883ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2043 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
47192025-05-08T22:05:38.883ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
47202025-05-08T22:05:38.883ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
47212025-05-08T22:05:38.884ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
47222025-05-08T22:05:38.886ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
47232025-05-08T22:05:38.886ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
47242025-05-08T22:05:39.050ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "8f13b764-6924-4143-b958-73289581d02f", "date": "Thu, 08 May 2025 22:05:38 GMT"} })
47252025-05-08T22:05:39.050ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch = Switch0
47262025-05-08T22:05:39.050ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
47272025-05-08T22:05:39.055ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "04eafb99-0145-4113-85bf-b37a82bcf86f", "content-length": "682", "date": "Thu, 08 May 2025 22:05:38 GMT"} })
47282025-05-08T22:05:39.055ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
47292025-05-08T22:05:39.056ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
47302025-05-08T22:05:39.060ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
47312025-05-08T22:05:39.060ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
47322025-05-08T22:05:39.063ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
47332025-05-08T22:05:39.064ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
47342025-05-08T22:05:39.065ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
47352025-05-08T22:05:39.066ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
47362025-05-08T22:05:39.073ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 791.778579ms iteration = 15 status = {}
47372025-05-08T22:05:41.937ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 293 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = d135ad92-8dac-4726-a5e6-079ad3fa2173 response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
47382025-05-08T22:05:48.269ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 44 reason = Timeout
47392025-05-08T22:05:48.269ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
47402025-05-08T22:05:48.269ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
47412025-05-08T22:05:48.271ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47422025-05-08T22:05:48.273ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
47432025-05-08T22:05:48.273ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
47442025-05-08T22:05:48.273ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
47452025-05-08T22:05:48.278ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
47462025-05-08T22:05:48.278ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
47472025-05-08T22:05:48.280ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47482025-05-08T22:05:48.281ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 44 reason = Timeout
47492025-05-08T22:05:48.281ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
47502025-05-08T22:05:48.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
47512025-05-08T22:05:48.283ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
47522025-05-08T22:05:48.283ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
47532025-05-08T22:05:48.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
47542025-05-08T22:05:48.298ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
47552025-05-08T22:05:48.298ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 28.684167ms iteration = 44 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
47562025-05-08T22:05:48.317ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
47572025-05-08T22:05:48.317ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
47582025-05-08T22:05:48.350ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 68.619134ms iteration = 44 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
47592025-05-08T22:05:51.937ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 289 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 53da334d-3b7f-4e47-9bfa-26eb3c60d054 response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
47602025-05-08T22:05:58.269ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 45 reason = Timeout
47612025-05-08T22:05:58.269ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
47622025-05-08T22:05:58.269ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
47632025-05-08T22:05:58.271ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47642025-05-08T22:05:58.274ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
47652025-05-08T22:05:58.274ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
47662025-05-08T22:05:58.274ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
47672025-05-08T22:05:58.276ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
47682025-05-08T22:05:58.276ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
47692025-05-08T22:05:58.278ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47702025-05-08T22:05:58.280ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
47712025-05-08T22:05:58.280ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
47722025-05-08T22:05:58.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
47732025-05-08T22:05:58.282ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 45 reason = Timeout
47742025-05-08T22:05:58.282ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
47752025-05-08T22:05:58.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
47762025-05-08T22:05:58.283ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
47772025-05-08T22:05:58.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
47782025-05-08T22:05:58.285ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.078404ms iteration = 45 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
47792025-05-08T22:05:58.288ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
47802025-05-08T22:05:58.288ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 19.141895ms iteration = 45 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
47812025-05-08T22:06:01.936ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 277 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 569a0e16-d362-4a49-b402-59bd66295ea1 response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
47822025-05-08T22:06:08.267ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 16 reason = Timeout
47832025-05-08T22:06:08.267ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 16 reason = Timeout
47842025-05-08T22:06:08.267ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
47852025-05-08T22:06:08.267ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
47862025-05-08T22:06:08.267ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
47872025-05-08T22:06:08.267ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
47882025-05-08T22:06:08.269ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 46 reason = Timeout
47892025-05-08T22:06:08.269ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 16 reason = Timeout
47902025-05-08T22:06:08.269ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
47912025-05-08T22:06:08.269ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
47922025-05-08T22:06:08.270ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1344, tv_nsec: 10392210 } })
47932025-05-08T22:06:08.270ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1344, tv_nsec: 10580742 } })
47942025-05-08T22:06:08.270ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1163
47952025-05-08T22:06:08.270ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 16 reason = Timeout
47962025-05-08T22:06:08.270ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
47972025-05-08T22:06:08.270ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163
47982025-05-08T22:06:08.271ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
47992025-05-08T22:06:08.271ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48002025-05-08T22:06:08.275ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
48012025-05-08T22:06:08.275ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
48022025-05-08T22:06:08.275ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
48032025-05-08T22:06:08.276ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48042025-05-08T22:06:08.277ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 7.973675ms iteration = 16 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
48052025-05-08T22:06:08.277ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
48062025-05-08T22:06:08.278ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
48072025-05-08T22:06:08.279ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48082025-05-08T22:06:08.279ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
48092025-05-08T22:06:08.279ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
48102025-05-08T22:06:08.279ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
48112025-05-08T22:06:08.281ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 46 reason = Timeout
48122025-05-08T22:06:08.282ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 16 reason = Timeout
48132025-05-08T22:06:08.282ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
48142025-05-08T22:06:08.282ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
48152025-05-08T22:06:08.282ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
48162025-05-08T22:06:08.282ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 16 reason = Timeout
48172025-05-08T22:06:08.282ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
48182025-05-08T22:06:08.282ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
48192025-05-08T22:06:08.282ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 16 reason = Timeout
48202025-05-08T22:06:08.282ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
48212025-05-08T22:06:08.282ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 16 reason = Timeout
48222025-05-08T22:06:08.282ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
48232025-05-08T22:06:08.282ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 16 reason = Timeout
48242025-05-08T22:06:08.282ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
48252025-05-08T22:06:08.282ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 17 reason = Timeout
48262025-05-08T22:06:08.282ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
48272025-05-08T22:06:08.282ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 16 reason = Timeout
48282025-05-08T22:06:08.282ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
48292025-05-08T22:06:08.282ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 16 reason = Timeout
48302025-05-08T22:06:08.282ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
48312025-05-08T22:06:08.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
48322025-05-08T22:06:08.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
48332025-05-08T22:06:08.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
48342025-05-08T22:06:08.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
48352025-05-08T22:06:08.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
48362025-05-08T22:06:08.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
48372025-05-08T22:06:08.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
48382025-05-08T22:06:08.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
48392025-05-08T22:06:08.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
48402025-05-08T22:06:08.282ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
48412025-05-08T22:06:08.282ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
48422025-05-08T22:06:08.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
48432025-05-08T22:06:08.284ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 16 reason = Timeout
48442025-05-08T22:06:08.284ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 16 reason = Timeout
48452025-05-08T22:06:08.284ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
48462025-05-08T22:06:08.284ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
48472025-05-08T22:06:08.284ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 16 reason = Timeout
48482025-05-08T22:06:08.284ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
48492025-05-08T22:06:08.284ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 16 reason = Timeout
48502025-05-08T22:06:08.284ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
48512025-05-08T22:06:08.284ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 16 reason = Timeout
48522025-05-08T22:06:08.284ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
48532025-05-08T22:06:08.284ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48542025-05-08T22:06:08.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
48552025-05-08T22:06:08.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
48562025-05-08T22:06:08.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
48572025-05-08T22:06:08.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
48582025-05-08T22:06:08.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
48592025-05-08T22:06:08.285ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48602025-05-08T22:06:08.286ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48612025-05-08T22:06:08.287ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
48622025-05-08T22:06:08.287ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
48632025-05-08T22:06:08.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
48642025-05-08T22:06:08.287ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
48652025-05-08T22:06:08.287ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
48662025-05-08T22:06:08.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
48672025-05-08T22:06:08.289ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48682025-05-08T22:06:08.289ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
48692025-05-08T22:06:08.289ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
48702025-05-08T22:06:08.289ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48712025-05-08T22:06:08.289ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
48722025-05-08T22:06:08.293ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
48732025-05-08T22:06:08.293ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
48742025-05-08T22:06:08.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
48752025-05-08T22:06:08.294ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
48762025-05-08T22:06:08.294ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
48772025-05-08T22:06:08.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
48782025-05-08T22:06:08.304ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
48792025-05-08T22:06:08.304ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
48802025-05-08T22:06:08.304ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
48812025-05-08T22:06:08.304ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
48822025-05-08T22:06:08.306ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1344, tv_nsec: 46105262 } })
48832025-05-08T22:06:08.306ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
48842025-05-08T22:06:08.331ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
48852025-05-08T22:06:08.331ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 61.536958ms iteration = 46 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
48862025-05-08T22:06:08.334ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
48872025-05-08T22:06:08.334ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
48882025-05-08T22:06:08.334ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
48892025-05-08T22:06:08.335ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
48902025-05-08T22:06:08.338ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
48912025-05-08T22:06:08.338ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
48922025-05-08T22:06:08.356ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
48932025-05-08T22:06:08.356ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 72.400054ms iteration = 16 status = {"errors":[],"garbage_collect_requested":[]}
48942025-05-08T22:06:08.356ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
48952025-05-08T22:06:08.357ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
48962025-05-08T22:06:08.357ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
48972025-05-08T22:06:08.357ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
48982025-05-08T22:06:08.358ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
48992025-05-08T22:06:08.359ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
49002025-05-08T22:06:08.360ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
49012025-05-08T22:06:08.363ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
49022025-05-08T22:06:08.363ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
49032025-05-08T22:06:08.365ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
49042025-05-08T22:06:08.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
49052025-05-08T22:06:08.367ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 83.2304ms iteration = 16 status = {"errors":[],"finish_invoked_ok":[]}
49062025-05-08T22:06:08.368ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 83.729906ms iteration = 16 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
49072025-05-08T22:06:08.369ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49082025-05-08T22:06:08.370ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 86.06059ms iteration = 16 status = {"errors":[],"requests_created_ok":[]}
49092025-05-08T22:06:08.371ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
49102025-05-08T22:06:08.371ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
49112025-05-08T22:06:08.371ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
49122025-05-08T22:06:08.371ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:552 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
49132025-05-08T22:06:08.371ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 89.773551ms iteration = 16 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
49142025-05-08T22:06:08.372ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
49152025-05-08T22:06:08.372ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
49162025-05-08T22:06:08.374ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 90.382566ms iteration = 16 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
49172025-05-08T22:06:08.375ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49182025-05-08T22:06:08.377ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
49192025-05-08T22:06:08.377ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
49202025-05-08T22:06:08.378ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
49212025-05-08T22:06:08.392ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
49222025-05-08T22:06:08.392ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
49232025-05-08T22:06:08.392ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f1f6e58-8af0-4018-9296-d726eb129c49", "content-length": "26", "date": "Thu, 08 May 2025 22:06:08 GMT"} })
49242025-05-08T22:06:08.392ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
49252025-05-08T22:06:08.393ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
49262025-05-08T22:06:08.393ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
49272025-05-08T22:06:08.393ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "db763301-7404-4fcc-8ad7-33e1eb08cd14", "content-length": "26", "date": "Thu, 08 May 2025 22:06:08 GMT"} })
49282025-05-08T22:06:08.393ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
49292025-05-08T22:06:08.394ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
49302025-05-08T22:06:08.394ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
49312025-05-08T22:06:08.397ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49322025-05-08T22:06:08.401ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49332025-05-08T22:06:08.402ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 2f2f2c32-8d58-4801-a669-9618eac0c3f7, lookup_type: ById(2f2f2c32-8d58-4801-a669-9618eac0c3f7) } result = Ok(())
49342025-05-08T22:06:08.402ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
49352025-05-08T22:06:08.402ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
49362025-05-08T22:06:08.403ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
49372025-05-08T22:06:08.403ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
49382025-05-08T22:06:08.404ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
49392025-05-08T22:06:08.405ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 123.756244ms iteration = 16 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
49402025-05-08T22:06:08.407ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 125.774845ms iteration = 46 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
49412025-05-08T22:06:08.409ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
49422025-05-08T22:06:08.409ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
49432025-05-08T22:06:08.409ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
49442025-05-08T22:06:08.409ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
49452025-05-08T22:06:08.421ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e zone_address = fd00:1122:3344:101::2
49462025-05-08T22:06:08.421ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
49472025-05-08T22:06:08.421ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8b086ad-2741-4410-bd79-cb62e4cf7970", "content-length": "26", "date": "Thu, 08 May 2025 22:06:08 GMT"} })
49482025-05-08T22:06:08.421ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1174 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
49492025-05-08T22:06:08.423ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
49502025-05-08T22:06:08.451ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
49512025-05-08T22:06:08.451ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
49522025-05-08T22:06:08.452ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
49532025-05-08T22:06:08.452ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
49542025-05-08T22:06:08.452ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc796def-f9b7-4683-84b0-4df526aedde5", "content-length": "372", "date": "Thu, 08 May 2025 22:06:08 GMT"} })
49552025-05-08T22:06:08.453ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
49562025-05-08T22:06:08.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
49572025-05-08T22:06:08.454ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 172.929739ms iteration = 16 status = 0
49582025-05-08T22:06:08.466ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 198.759056ms iteration = 16 status = {}
49592025-05-08T22:06:08.468ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
49602025-05-08T22:06:08.468ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
49612025-05-08T22:06:08.468ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "cb6f0ee6-146e-4240-beab-931e9dc1418c", "date": "Thu, 08 May 2025 22:06:08 GMT"} })
49622025-05-08T22:06:08.468ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
49632025-05-08T22:06:08.469ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
49642025-05-08T22:06:08.470ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
49652025-05-08T22:06:08.470ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
49662025-05-08T22:06:08.473ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
49672025-05-08T22:06:08.473ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
49682025-05-08T22:06:08.473ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
49692025-05-08T22:06:08.474ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
49702025-05-08T22:06:08.475ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 207.48229ms iteration = 16 status = null
49712025-05-08T22:06:08.476ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
49722025-05-08T22:06:08.476ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
49732025-05-08T22:06:08.477ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "9bf12d3f-7b35-4453-8ba4-5d466476f422", "date": "Thu, 08 May 2025 22:06:08 GMT"} })
49742025-05-08T22:06:08.479ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 197.26853ms iteration = 16 status = {}
49752025-05-08T22:06:08.486ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
49762025-05-08T22:06:08.486ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
49772025-05-08T22:06:08.486ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 204.061562ms iteration = 16 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
49782025-05-08T22:06:08.487ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:362
49792025-05-08T22:06:08.487ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 216.401115ms iteration = 16 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
49802025-05-08T22:06:08.587ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
49812025-05-08T22:06:08.588ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
49822025-05-08T22:06:08.597ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: cd0b930c-534b-46ad-ac3b-2eaa94b0bf2c, rack_id: 64f3d6f4-89b8-48aa-a622-b684e261667e, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(65f83a36-18fb-49b7-9914-87c8517d200e) } rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
49832025-05-08T22:06:08.597ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
49842025-05-08T22:06:08.598ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
49852025-05-08T22:06:08.609ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1833 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
49862025-05-08T22:06:08.609ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1860 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 65f83a36-18fb-49b7-9914-87c8517d200e
49872025-05-08T22:06:08.610ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
49882025-05-08T22:06:08.610ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
49892025-05-08T22:06:08.610ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
49902025-05-08T22:06:08.610ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
49912025-05-08T22:06:08.610ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
49922025-05-08T22:06:08.610ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2007 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
49932025-05-08T22:06:08.610ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
49942025-05-08T22:06:08.610ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2043 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
49952025-05-08T22:06:08.610ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
49962025-05-08T22:06:08.610ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
49972025-05-08T22:06:08.611ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
49982025-05-08T22:06:08.611ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 329.17511ms iteration = 17 status = {"physical_disks_added":0}
49992025-05-08T22:06:08.612ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
50002025-05-08T22:06:08.612ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
50012025-05-08T22:06:08.614ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
50022025-05-08T22:06:08.614ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
50032025-05-08T22:06:08.664ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50042025-05-08T22:06:08.666ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
50052025-05-08T22:06:08.667ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
50062025-05-08T22:06:08.667ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
50072025-05-08T22:06:08.669ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50082025-05-08T22:06:08.672ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
50092025-05-08T22:06:08.672ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
50102025-05-08T22:06:08.672ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
50112025-05-08T22:06:08.713ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
50122025-05-08T22:06:08.768ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "979863c3-088c-40d4-a8cf-4003af129f85", "date": "Thu, 08 May 2025 22:06:08 GMT"} })
50132025-05-08T22:06:08.768ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch = Switch0
50142025-05-08T22:06:08.768ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
50152025-05-08T22:06:08.769ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8970418-d79f-41af-b2d0-a8dbe1d2e53b", "content-length": "691", "date": "Thu, 08 May 2025 22:06:08 GMT"} })
50162025-05-08T22:06:08.769ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
50172025-05-08T22:06:08.769ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
50182025-05-08T22:06:08.769ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
50192025-05-08T22:06:08.769ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "6475677b-14bb-413b-afe9-0facab5bfc71", "content-length": "682", "date": "Thu, 08 May 2025 22:06:08 GMT"} })
50202025-05-08T22:06:08.769ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
50212025-05-08T22:06:08.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
50222025-05-08T22:06:08.770ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 488.809758ms iteration = 16 status = {}
50232025-05-08T22:06:08.775ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
50242025-05-08T22:06:08.775ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
50252025-05-08T22:06:08.779ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
50262025-05-08T22:06:08.779ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
50272025-05-08T22:06:08.782ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
50282025-05-08T22:06:08.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
50292025-05-08T22:06:08.791ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 509.008905ms iteration = 16 status = {}
50302025-05-08T22:06:11.937ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 282 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 4201ab36-24cf-4739-8ab0-0714174e251e response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
50312025-05-08T22:06:18.269ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 47 reason = Timeout
50322025-05-08T22:06:18.269ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
50332025-05-08T22:06:18.269ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
50342025-05-08T22:06:18.271ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50352025-05-08T22:06:18.274ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
50362025-05-08T22:06:18.274ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
50372025-05-08T22:06:18.274ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
50382025-05-08T22:06:18.276ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
50392025-05-08T22:06:18.277ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
50402025-05-08T22:06:18.278ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50412025-05-08T22:06:18.280ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
50422025-05-08T22:06:18.281ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
50432025-05-08T22:06:18.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
50442025-05-08T22:06:18.282ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 47 reason = Timeout
50452025-05-08T22:06:18.282ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
50462025-05-08T22:06:18.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
50472025-05-08T22:06:18.284ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
50482025-05-08T22:06:18.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
50492025-05-08T22:06:18.292ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
50502025-05-08T22:06:18.292ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.907116ms iteration = 47 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
50512025-05-08T22:06:18.321ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 38.456212ms iteration = 47 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
50522025-05-08T22:06:21.929ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::d]:60004
50532025-05-08T22:06:21.930ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60004 req_id = a16de22f-7840-400d-9596-260786d9eb1d roles = RoleSet { roles: {} } uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?sort_by=id_ascending
50542025-05-08T22:06:21.930ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60004 req_id = a16de22f-7840-400d-9596-260786d9eb1d resource = Database result = Ok(()) uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?sort_by=id_ascending
50552025-05-08T22:06:21.933ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3162 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60004 req_id = a16de22f-7840-400d-9596-260786d9eb1d response_code = 200 uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?sort_by=id_ascending
50562025-05-08T22:06:21.933ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60004 req_id = 4492adc3-c77c-46f2-a4a2-97d2cba0fb82 roles = RoleSet { roles: {} } uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZGQ4MjYzNC0yODQ0LTRmOTctOTc0Ni0wNzA5OTA0NGMxOWQifX0%3D
50572025-05-08T22:06:21.933ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60004 req_id = 4492adc3-c77c-46f2-a4a2-97d2cba0fb82 resource = Database result = Ok(()) uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZGQ4MjYzNC0yODQ0LTRmOTctOTc0Ni0wNzA5OTA0NGMxOWQifX0%3D
50582025-05-08T22:06:21.935ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2566 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60004 req_id = 4492adc3-c77c-46f2-a4a2-97d2cba0fb82 response_code = 200 uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZGQ4MjYzNC0yODQ0LTRmOTctOTc0Ni0wNzA5OTA0NGMxOWQifX0%3D
50592025-05-08T22:06:21.937ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 210 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 4afa1606-b42e-4671-a867-45f9a3f555a3 response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
50602025-05-08T22:06:28.269ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 48 reason = Timeout
50612025-05-08T22:06:28.269ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
50622025-05-08T22:06:28.269ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
50632025-05-08T22:06:28.271ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50642025-05-08T22:06:28.274ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
50652025-05-08T22:06:28.274ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
50662025-05-08T22:06:28.274ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
50672025-05-08T22:06:28.276ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
50682025-05-08T22:06:28.276ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
50692025-05-08T22:06:28.277ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50702025-05-08T22:06:28.280ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
50712025-05-08T22:06:28.280ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
50722025-05-08T22:06:28.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
50732025-05-08T22:06:28.282ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 48 reason = Timeout
50742025-05-08T22:06:28.282ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
50752025-05-08T22:06:28.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
50762025-05-08T22:06:28.283ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
50772025-05-08T22:06:28.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
50782025-05-08T22:06:28.292ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
50792025-05-08T22:06:28.292ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 23.037856ms iteration = 48 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
50802025-05-08T22:06:28.323ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 41.13576ms iteration = 48 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
50812025-05-08T22:06:31.937ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 347 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = cd5581df-1158-45c2-b628-5fde095acd2c response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
50822025-05-08T22:06:38.267ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 17 reason = Timeout
50832025-05-08T22:06:38.267ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
50842025-05-08T22:06:38.267ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 11 reason = Timeout
50852025-05-08T22:06:38.267ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 9 reason = Timeout
50862025-05-08T22:06:38.267ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
50872025-05-08T22:06:38.267ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 9 reason = Timeout
50882025-05-08T22:06:38.267ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
50892025-05-08T22:06:38.267ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 9 reason = Timeout
50902025-05-08T22:06:38.267ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
50912025-05-08T22:06:38.267ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 9 reason = Timeout
50922025-05-08T22:06:38.267ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
50932025-05-08T22:06:38.267ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 17 reason = Timeout
50942025-05-08T22:06:38.267ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
50952025-05-08T22:06:38.267ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
50962025-05-08T22:06:38.267ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 9 reason = Timeout
50972025-05-08T22:06:38.267ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 11 reason = Timeout
50982025-05-08T22:06:38.267ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 9 reason = Timeout
50992025-05-08T22:06:38.267ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-05-08 21:56:38.267229866 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
51002025-05-08T22:06:38.267ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
51012025-05-08T22:06:38.267ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
51022025-05-08T22:06:38.267ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
51032025-05-08T22:06:38.267ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
51042025-05-08T22:06:38.267ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
51052025-05-08T22:06:38.267ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
51062025-05-08T22:06:38.267ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
51072025-05-08T22:06:38.267ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("0ccabf41-81cd-45e7-bb26-335edaa2ee47.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("38a08dd6-6a75-4558-b4f8-df853f4d03d9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("38a08dd6-6a75-4558-b4f8-df853f4d03d9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::9))) }], valid_until: Instant { tv_sec: 1374, tv_nsec: 7499809 } })
51082025-05-08T22:06:38.268ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("228ed1c1-3cc1-4055-aae9-5595ad856b86.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("8b245480-f91d-406b-bb08-850e1c3e3fc5.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c8f20f67-ea63-4dbc-b0f1-c30876daff82.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c8f20f67-ea63-4dbc-b0f1-c30876daff82.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 1374, tv_nsec: 7722281 } })
51092025-05-08T22:06:38.268ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } dns_group = external
51102025-05-08T22:06:38.268ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.271874ms iteration = 9 status = {"addresses":["[fd00:1122:3344:101::8]:5353","[fd00:1122:3344:101::9]:5353"]}
51112025-05-08T22:06:38.268ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
51122025-05-08T22:06:38.268ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.399495ms iteration = 9 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:1::1]:5353"]}
51132025-05-08T22:06:38.269ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 10 reason = Timeout
51142025-05-08T22:06:38.269ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 49 reason = Timeout
51152025-05-08T22:06:38.269ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:85 reason = blueprint disabled target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2
51162025-05-08T22:06:38.269ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 17 reason = Timeout
51172025-05-08T22:06:38.269ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 13.51\u{b5}s iteration = 10 status = {"enabled":false,"target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2"}
51182025-05-08T22:06:38.269ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
51192025-05-08T22:06:38.269ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
51202025-05-08T22:06:38.270ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
51212025-05-08T22:06:38.270ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1374, tv_nsec: 9985526 } })
51222025-05-08T22:06:38.270ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
51232025-05-08T22:06:38.270ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1163
51242025-05-08T22:06:38.270ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-05-08 21:56:38.267229866 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
51252025-05-08T22:06:38.270ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 3.387556ms iteration = 9 status = {"expiration":"2025-05-08T21:56:38.267229866Z","pruned":[]}
51262025-05-08T22:06:38.270ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
51272025-05-08T22:06:38.270ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51282025-05-08T22:06:38.270ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
51292025-05-08T22:06:38.271ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1374, tv_nsec: 10725493 } })
51302025-05-08T22:06:38.271ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 17 reason = Timeout
51312025-05-08T22:06:38.271ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
51322025-05-08T22:06:38.271ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163
51332025-05-08T22:06:38.271ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
51342025-05-08T22:06:38.272ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51352025-05-08T22:06:38.272ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
51362025-05-08T22:06:38.272ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
51372025-05-08T22:06:38.272ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
51382025-05-08T22:06:38.274ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 5.068364ms iteration = 17 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
51392025-05-08T22:06:38.275ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51402025-05-08T22:06:38.276ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51412025-05-08T22:06:38.276ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
51422025-05-08T22:06:38.276ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
51432025-05-08T22:06:38.276ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
51442025-05-08T22:06:38.276ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
51452025-05-08T22:06:38.276ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
51462025-05-08T22:06:38.278ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51472025-05-08T22:06:38.278ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
51482025-05-08T22:06:38.279ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
51492025-05-08T22:06:38.279ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
51502025-05-08T22:06:38.279ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
51512025-05-08T22:06:38.279ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
51522025-05-08T22:06:38.280ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
51532025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
51542025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
51552025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
51562025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 9 reason = Timeout
51572025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 49 reason = Timeout
51582025-05-08T22:06:38.284ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:53
51592025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
51602025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
51612025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
51622025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 17 reason = Timeout
51632025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
51642025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
51652025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
51662025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 17 reason = Timeout
51672025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
51682025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
51692025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 18 reason = Timeout
51702025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
51712025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
51722025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 17 reason = Timeout
51732025-05-08T22:06:38.284ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
51742025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
51752025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
51762025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 9 reason = Timeout
51772025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
51782025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
51792025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 17 reason = Timeout
51802025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
51812025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
51822025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 17 reason = Timeout
51832025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
51842025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
51852025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 17 reason = Timeout
51862025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
51872025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
51882025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 17 reason = Timeout
51892025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
51902025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
51912025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): activating background_task = webhook_deliverator iteration = 9 reason = Timeout
51922025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} }
51932025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
51942025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 9 reason = Timeout
51952025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
51962025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 17 reason = Timeout
51972025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
51982025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 17 reason = Timeout
51992025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
52002025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 17 reason = Timeout
52012025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
52022025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database result = Ok(())
52032025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 17 reason = Timeout
52042025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 9 reason = Timeout
52052025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): activating background_task = webhook_dispatcher iteration = 9 reason = Timeout
52062025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
52072025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
52082025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): roles background_task = webhook_dispatcher roles = RoleSet { roles: {} }
52092025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 17 reason = Timeout
52102025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
52112025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
52122025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
52132025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
52142025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database result = Ok(())
52152025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
52162025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
52172025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
52182025-05-08T22:06:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
52192025-05-08T22:06:38.286ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
52202025-05-08T22:06:38.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
52212025-05-08T22:06:38.286ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52222025-05-08T22:06:38.287ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
52232025-05-08T22:06:38.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
52242025-05-08T22:06:38.289ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
52252025-05-08T22:06:38.289ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
52262025-05-08T22:06:38.289ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
52272025-05-08T22:06:38.289ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
52282025-05-08T22:06:38.289ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
52292025-05-08T22:06:38.289ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
52302025-05-08T22:06:38.290ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
52312025-05-08T22:06:38.291ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
52322025-05-08T22:06:38.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
52332025-05-08T22:06:38.293ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52342025-05-08T22:06:38.293ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52352025-05-08T22:06:38.294ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52362025-05-08T22:06:38.295ZDEBGnexus (BackgroundTasks): activation complete background_task = webhook_deliverator elapsed = 11.2948ms iteration = 9 status = {"by_rx":{},"error":null}
52372025-05-08T22:06:38.296ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
52382025-05-08T22:06:38.296ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
52392025-05-08T22:06:38.296ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
52402025-05-08T22:06:38.298ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
52412025-05-08T22:06:38.298ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
52422025-05-08T22:06:38.298ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
52432025-05-08T22:06:38.299ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
52442025-05-08T22:06:38.299ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
52452025-05-08T22:06:38.299ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
52462025-05-08T22:06:38.303ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 19.171905ms iteration = 17 status = {"errors":[],"garbage_collect_requested":[]}
52472025-05-08T22:06:38.306ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
52482025-05-08T22:06:38.306ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 22.308629ms iteration = 9 status = {"errors":[],"found_port_ok":[]}
52492025-05-08T22:06:38.306ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
52502025-05-08T22:06:38.308ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
52512025-05-08T22:06:38.308ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
52522025-05-08T22:06:38.308ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 24.029737ms iteration = 9 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
52532025-05-08T22:06:38.333ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:552 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
52542025-05-08T22:06:38.333ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 49.406429ms iteration = 17 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
52552025-05-08T22:06:38.338ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1374, tv_nsec: 78263336 } })
52562025-05-08T22:06:38.338ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52572025-05-08T22:06:38.338ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
52582025-05-08T22:06:38.341ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
52592025-05-08T22:06:38.341ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
52602025-05-08T22:06:38.343ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
52612025-05-08T22:06:38.343ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
52622025-05-08T22:06:38.343ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
52632025-05-08T22:06:38.343ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
52642025-05-08T22:06:38.343ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
52652025-05-08T22:06:38.344ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
52662025-05-08T22:06:38.348ZDEBGnexus (BackgroundTasks): roles background_task = webhook_dispatcher roles = RoleSet { roles: {} }
52672025-05-08T22:06:38.348ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database result = Ok(())
52682025-05-08T22:06:38.353ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52692025-05-08T22:06:38.354ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
52702025-05-08T22:06:38.354ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
52712025-05-08T22:06:38.356ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52722025-05-08T22:06:38.356ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
52732025-05-08T22:06:38.356ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
52742025-05-08T22:06:38.356ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
52752025-05-08T22:06:38.356ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
52762025-05-08T22:06:38.356ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52772025-05-08T22:06:38.357ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
52782025-05-08T22:06:38.358ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
52792025-05-08T22:06:38.358ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
52802025-05-08T22:06:38.358ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
52812025-05-08T22:06:38.360ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
52822025-05-08T22:06:38.360ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
52832025-05-08T22:06:38.360ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
52842025-05-08T22:06:38.361ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
52852025-05-08T22:06:38.361ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
52862025-05-08T22:06:38.361ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52872025-05-08T22:06:38.361ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
52882025-05-08T22:06:38.365ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
52892025-05-08T22:06:38.365ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
52902025-05-08T22:06:38.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
52912025-05-08T22:06:38.367ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52922025-05-08T22:06:38.368ZDEBGnexus (BackgroundTasks): activation complete background_task = webhook_dispatcher elapsed = 84.219511ms iteration = 9 status = {"dispatched":[],"errors":[],"glob_version":"140.0.0","globs_reprocessed":{},"no_receivers":[]}
52932025-05-08T22:06:38.369ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
52942025-05-08T22:06:38.369ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
52952025-05-08T22:06:38.369ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
52962025-05-08T22:06:38.370ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
52972025-05-08T22:06:38.371ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
52982025-05-08T22:06:38.371ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
52992025-05-08T22:06:38.372ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
53002025-05-08T22:06:38.372ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 89.111363ms iteration = 49 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
53012025-05-08T22:06:38.373ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
53022025-05-08T22:06:38.373ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
53032025-05-08T22:06:38.374ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 89.997282ms iteration = 17 status = {"errors":[],"finish_invoked_ok":[]}
53042025-05-08T22:06:38.376ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
53052025-05-08T22:06:38.377ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
53062025-05-08T22:06:38.378ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 94.214767ms iteration = 17 status = {"errors":[],"requests_created_ok":[]}
53072025-05-08T22:06:38.379ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53082025-05-08T22:06:38.379ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53092025-05-08T22:06:38.379ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
53102025-05-08T22:06:38.379ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 110.492941ms iteration = 49 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
53112025-05-08T22:06:38.382ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53122025-05-08T22:06:38.382ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
53132025-05-08T22:06:38.382ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
53142025-05-08T22:06:38.382ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 99.624995ms iteration = 17 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
53152025-05-08T22:06:38.382ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
53162025-05-08T22:06:38.383ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
53172025-05-08T22:06:38.383ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
53182025-05-08T22:06:38.384ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
53192025-05-08T22:06:38.386ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
53202025-05-08T22:06:38.386ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
53212025-05-08T22:06:38.388ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
53222025-05-08T22:06:38.406ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
53232025-05-08T22:06:38.407ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:137 found = 0 not_ready_to_be_deleted = 0
53242025-05-08T22:06:38.407ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 124.129637ms iteration = 9 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
53252025-05-08T22:06:38.409ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
53262025-05-08T22:06:38.410ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
53272025-05-08T22:06:38.411ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
53282025-05-08T22:06:38.411ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
53292025-05-08T22:06:38.413ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
53302025-05-08T22:06:38.414ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
53312025-05-08T22:06:38.414ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
53322025-05-08T22:06:38.421ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53332025-05-08T22:06:38.425ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
53342025-05-08T22:06:38.425ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = d8128696-db5a-4b88-914a-69b28494e50e dns_zone_name = oxide.test found_so_far = 0 version = 2
53352025-05-08T22:06:38.425ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
53362025-05-08T22:06:38.425ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
53372025-05-08T22:06:38.425ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 2f2f2c32-8d58-4801-a669-9618eac0c3f7, lookup_type: ById(2f2f2c32-8d58-4801-a669-9618eac0c3f7) } result = Ok(())
53382025-05-08T22:06:38.425ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
53392025-05-08T22:06:38.425ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
53402025-05-08T22:06:38.426ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53412025-05-08T22:06:38.430ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
53422025-05-08T22:06:38.430ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
53432025-05-08T22:06:38.430ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
53442025-05-08T22:06:38.433ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = d8128696-db5a-4b88-914a-69b28494e50e dns_zone_name = oxide.test found_so_far = 1 version = 2
53452025-05-08T22:06:38.433ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
53462025-05-08T22:06:38.433ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-05-08 21:59:21.759368 UTC dns_group = external generation = 2
53472025-05-08T22:06:38.433ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 166.093546ms iteration = 9 status = {"generation":2}
53482025-05-08T22:06:38.433ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 149.66239ms iteration = 17 status = 0
53492025-05-08T22:06:38.433ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
53502025-05-08T22:06:38.433ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 7586992a-6fc5-4073-8962-728778f16aa5 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
53512025-05-08T22:06:38.433ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
53522025-05-08T22:06:38.434ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
53532025-05-08T22:06:38.435ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 150.760233ms iteration = 17 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
53542025-05-08T22:06:38.436ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53552025-05-08T22:06:38.437ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
53562025-05-08T22:06:38.437ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
53572025-05-08T22:06:38.437ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 154.740955ms iteration = 17 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
53582025-05-08T22:06:38.440ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
53592025-05-08T22:06:38.440ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
53602025-05-08T22:06:38.440ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
53612025-05-08T22:06:38.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
53622025-05-08T22:06:38.443ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 7586992a-6fc5-4073-8962-728778f16aa5 dns_zone_name = control-plane.oxide.internal found_so_far = 55 version = 1
53632025-05-08T22:06:38.443ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
53642025-05-08T22:06:38.443ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-05-08 21:59:21.182334 UTC dns_group = internal generation = 1
53652025-05-08T22:06:38.443ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 176.624688ms iteration = 9 status = {"generation":1}
53662025-05-08T22:06:38.445ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:362
53672025-05-08T22:06:38.445ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 174.005441ms iteration = 17 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
53682025-05-08T22:06:38.449ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53692025-05-08T22:06:38.451ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
53702025-05-08T22:06:38.451ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
53712025-05-08T22:06:38.452ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
53722025-05-08T22:06:38.452ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-05-08T21:58:35.560911Z, time_modified: 2025-05-08T21:58:35.560911Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, db_silo: Silo { identity: SiloIdentity { id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-05-08T21:59:21.709488Z, time_modified: 2025-05-08T21:59:21.709488Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, db_silo: Silo { identity: SiloIdentity { id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-05-08T21:59:21.709488Z, time_modified: 2025-05-08T21:59:21.709488Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47" }] }) }
53732025-05-08T22:06:38.452ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 185.695736ms iteration = 9 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"7137ee58-6aa7-4199-a248-feadf3b8bafb","tls_certs":["37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47"]}},"default_endpoint":{"silo_id":"7137ee58-6aa7-4199-a248-feadf3b8bafb","tls_certs":["37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
53742025-05-08T22:06:38.456ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 173.183582ms iteration = 18 status = {"physical_disks_added":0}
53752025-05-08T22:06:38.471ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 187.219842ms iteration = 17 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
53762025-05-08T22:06:38.471ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
53772025-05-08T22:06:38.471ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
53782025-05-08T22:06:38.482ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
53792025-05-08T22:06:38.483ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
53802025-05-08T22:06:38.509ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e zone_address = fd00:1122:3344:101::2
53812025-05-08T22:06:38.509ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
53822025-05-08T22:06:38.509ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "83774cda-fb93-49c8-ad02-d8ce79221bbd", "content-length": "26", "date": "Thu, 08 May 2025 22:06:38 GMT"} })
53832025-05-08T22:06:38.509ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1174 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
53842025-05-08T22:06:38.511ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
53852025-05-08T22:06:38.550ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
53862025-05-08T22:06:38.550ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
53872025-05-08T22:06:38.550ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2efc2f4-262f-4c7b-af38-b1eb63c7e71f", "content-length": "26", "date": "Thu, 08 May 2025 22:06:38 GMT"} })
53882025-05-08T22:06:38.551ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
53892025-05-08T22:06:38.553ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
53902025-05-08T22:06:38.562ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
53912025-05-08T22:06:38.563ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
53922025-05-08T22:06:38.566ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53932025-05-08T22:06:38.569ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
53942025-05-08T22:06:38.569ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
53952025-05-08T22:06:38.569ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
53962025-05-08T22:06:38.572ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53972025-05-08T22:06:38.575ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
53982025-05-08T22:06:38.575ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
53992025-05-08T22:06:38.575ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
54002025-05-08T22:06:38.630ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
54012025-05-08T22:06:38.636ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
54022025-05-08T22:06:38.636ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "2cef491f-afc9-44bd-b00a-36f2677db951", "date": "Thu, 08 May 2025 22:06:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
54032025-05-08T22:06:38.636ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
54042025-05-08T22:06:38.638ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "d5e3a514-9aaf-4402-97bd-e7282b91a099", "date": "Thu, 08 May 2025 22:06:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
54052025-05-08T22:06:38.638ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
54062025-05-08T22:06:38.639ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
54072025-05-08T22:06:38.639ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 372.688537ms iteration = 11 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
54082025-05-08T22:06:38.640ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
54092025-05-08T22:06:38.640ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
54102025-05-08T22:06:38.640ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "6934b143-1b74-4a67-8103-559e094ac3a4", "content-length": "26", "date": "Thu, 08 May 2025 22:06:38 GMT"} })
54112025-05-08T22:06:38.640ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
54122025-05-08T22:06:38.642ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
54132025-05-08T22:06:38.678ZINFOnexus (BackgroundTasks): instance reincarnation completed successfully background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:113 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0
54142025-05-08T22:06:38.678ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 394.6022ms iteration = 9 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
54152025-05-08T22:06:38.745ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
54162025-05-08T22:06:38.745ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
54172025-05-08T22:06:38.746ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4ba9103-c497-40a2-8a10-93690abcd5fb", "content-length": "372", "date": "Thu, 08 May 2025 22:06:38 GMT"} })
54182025-05-08T22:06:38.746ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
54192025-05-08T22:06:38.746ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
54202025-05-08T22:06:38.763ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
54212025-05-08T22:06:38.763ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
54222025-05-08T22:06:38.763ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "e4def66e-5044-476a-9b9a-ab3d97fca2a3", "date": "Thu, 08 May 2025 22:06:38 GMT"} })
54232025-05-08T22:06:38.763ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
54242025-05-08T22:06:38.763ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
54252025-05-08T22:06:38.766ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
54262025-05-08T22:06:38.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
54272025-05-08T22:06:38.798ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 531.633467ms iteration = 17 status = {}
54282025-05-08T22:06:38.808ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
54292025-05-08T22:06:38.808ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d4b10fa-434a-4436-bdaf-f5e3b54c46a1", "content-length": "691", "date": "Thu, 08 May 2025 22:06:38 GMT"} })
54302025-05-08T22:06:38.808ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
54312025-05-08T22:06:38.808ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
54322025-05-08T22:06:38.808ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
54332025-05-08T22:06:38.809ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 526.34579ms iteration = 17 status = {}
54342025-05-08T22:06:38.811ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
54352025-05-08T22:06:38.811ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
54362025-05-08T22:06:38.814ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
54372025-05-08T22:06:38.814ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
54382025-05-08T22:06:38.815ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "e8499996-d151-4319-b29f-655bd69c4972", "date": "Thu, 08 May 2025 22:06:38 GMT"} })
54392025-05-08T22:06:38.816ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 533.448436ms iteration = 17 status = {}
54402025-05-08T22:06:38.847ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
54412025-05-08T22:06:38.848ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
54422025-05-08T22:06:38.849ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 582.662722ms iteration = 17 status = null
54432025-05-08T22:06:38.855ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
54442025-05-08T22:06:38.856ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "abb0fdb1-12ef-4071-8e8b-7155031c02da", "date": "Thu, 08 May 2025 22:06:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
54452025-05-08T22:06:38.856ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
54462025-05-08T22:06:38.857ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "9f380006-a551-4b68-931d-65176ae4bdd0", "date": "Thu, 08 May 2025 22:06:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
54472025-05-08T22:06:38.857ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
54482025-05-08T22:06:38.858ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "268a7a1f-5289-4575-9613-904b3f13395c", "date": "Thu, 08 May 2025 22:06:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
54492025-05-08T22:06:38.858ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
54502025-05-08T22:06:38.860ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
54512025-05-08T22:06:38.860ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 593.77555ms iteration = 11 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
54522025-05-08T22:06:38.889ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
54532025-05-08T22:06:38.890ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
54542025-05-08T22:06:38.892ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: cd0b930c-534b-46ad-ac3b-2eaa94b0bf2c, rack_id: 64f3d6f4-89b8-48aa-a622-b684e261667e, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(65f83a36-18fb-49b7-9914-87c8517d200e) } rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
54552025-05-08T22:06:38.892ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
54562025-05-08T22:06:38.892ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
54572025-05-08T22:06:38.904ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1833 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
54582025-05-08T22:06:38.904ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1860 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 65f83a36-18fb-49b7-9914-87c8517d200e
54592025-05-08T22:06:38.904ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
54602025-05-08T22:06:38.904ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
54612025-05-08T22:06:38.904ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
54622025-05-08T22:06:38.904ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
54632025-05-08T22:06:38.904ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
54642025-05-08T22:06:38.904ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2007 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
54652025-05-08T22:06:38.904ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
54662025-05-08T22:06:38.904ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2043 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
54672025-05-08T22:06:38.905ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
54682025-05-08T22:06:38.905ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
54692025-05-08T22:06:38.905ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
54702025-05-08T22:06:38.908ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
54712025-05-08T22:06:38.908ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
54722025-05-08T22:06:39.076ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "fae5a2c5-4476-4c45-bba8-1a2c46650ca0", "date": "Thu, 08 May 2025 22:06:38 GMT"} })
54732025-05-08T22:06:39.076ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch = Switch0
54742025-05-08T22:06:39.076ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
54752025-05-08T22:06:39.076ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "427fa6ff-662d-486a-94e5-202d22c313ed", "content-length": "682", "date": "Thu, 08 May 2025 22:06:38 GMT"} })
54762025-05-08T22:06:39.076ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
54772025-05-08T22:06:39.076ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
54782025-05-08T22:06:39.081ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
54792025-05-08T22:06:39.081ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
54802025-05-08T22:06:39.084ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
54812025-05-08T22:06:39.084ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
54822025-05-08T22:06:39.086ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
54832025-05-08T22:06:39.087ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
54842025-05-08T22:06:39.094ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 811.122246ms iteration = 17 status = {}
54852025-05-08T22:06:41.937ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 332 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = b22165e4-8c17-490d-b172-19bc312a9b2b response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
54862025-05-08T22:06:48.269ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 50 reason = Timeout
54872025-05-08T22:06:48.270ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
54882025-05-08T22:06:48.270ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
54892025-05-08T22:06:48.271ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54902025-05-08T22:06:48.274ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
54912025-05-08T22:06:48.274ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
54922025-05-08T22:06:48.274ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
54932025-05-08T22:06:48.277ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
54942025-05-08T22:06:48.277ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
54952025-05-08T22:06:48.278ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54962025-05-08T22:06:48.281ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
54972025-05-08T22:06:48.281ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
54982025-05-08T22:06:48.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
54992025-05-08T22:06:48.282ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 50 reason = Timeout
55002025-05-08T22:06:48.282ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
55012025-05-08T22:06:48.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
55022025-05-08T22:06:48.293ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
55032025-05-08T22:06:48.293ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 23.36969ms iteration = 50 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
55042025-05-08T22:06:48.315ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
55052025-05-08T22:06:48.315ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
55062025-05-08T22:06:48.346ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 63.793242ms iteration = 50 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
55072025-05-08T22:06:51.937ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 285 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = e14c1aca-7fcd-4614-9ab4-d245299aa2d3 response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
55082025-05-08T22:06:58.269ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 51 reason = Timeout
55092025-05-08T22:06:58.269ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
55102025-05-08T22:06:58.269ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
55112025-05-08T22:06:58.271ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55122025-05-08T22:06:58.274ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
55132025-05-08T22:06:58.274ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
55142025-05-08T22:06:58.274ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
55152025-05-08T22:06:58.275ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
55162025-05-08T22:06:58.276ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
55172025-05-08T22:06:58.278ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55182025-05-08T22:06:58.281ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
55192025-05-08T22:06:58.281ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
55202025-05-08T22:06:58.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
55212025-05-08T22:06:58.282ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 51 reason = Timeout
55222025-05-08T22:06:58.282ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
55232025-05-08T22:06:58.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
55242025-05-08T22:06:58.284ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
55252025-05-08T22:06:58.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
55262025-05-08T22:06:58.290ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
55272025-05-08T22:06:58.290ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.122026ms iteration = 51 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
55282025-05-08T22:06:58.316ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 33.494578ms iteration = 51 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
55292025-05-08T22:07:01.938ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 301 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 05eb6fd4-16fd-46b1-a796-2dbe3b77cf69 response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
55302025-05-08T22:07:08.267ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 18 reason = Timeout
55312025-05-08T22:07:08.267ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
55322025-05-08T22:07:08.267ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 18 reason = Timeout
55332025-05-08T22:07:08.267ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
55342025-05-08T22:07:08.268ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
55352025-05-08T22:07:08.268ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
55362025-05-08T22:07:08.270ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 52 reason = Timeout
55372025-05-08T22:07:08.270ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 18 reason = Timeout
55382025-05-08T22:07:08.270ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
55392025-05-08T22:07:08.270ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
55402025-05-08T22:07:08.270ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1404, tv_nsec: 10049866 } })
55412025-05-08T22:07:08.270ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1404, tv_nsec: 10196808 } })
55422025-05-08T22:07:08.270ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163
55432025-05-08T22:07:08.271ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1163
55442025-05-08T22:07:08.272ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55452025-05-08T22:07:08.272ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 18 reason = Timeout
55462025-05-08T22:07:08.272ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
55472025-05-08T22:07:08.272ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
55482025-05-08T22:07:08.273ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55492025-05-08T22:07:08.274ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.948622ms iteration = 18 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
55502025-05-08T22:07:08.275ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
55512025-05-08T22:07:08.275ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
55522025-05-08T22:07:08.275ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
55532025-05-08T22:07:08.275ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
55542025-05-08T22:07:08.275ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
55552025-05-08T22:07:08.276ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
55562025-05-08T22:07:08.276ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
55572025-05-08T22:07:08.276ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
55582025-05-08T22:07:08.283ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 52 reason = Timeout
55592025-05-08T22:07:08.283ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
55602025-05-08T22:07:08.283ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 18 reason = Timeout
55612025-05-08T22:07:08.283ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 18 reason = Timeout
55622025-05-08T22:07:08.283ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
55632025-05-08T22:07:08.283ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
55642025-05-08T22:07:08.283ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
55652025-05-08T22:07:08.283ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 18 reason = Timeout
55662025-05-08T22:07:08.283ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
55672025-05-08T22:07:08.283ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 18 reason = Timeout
55682025-05-08T22:07:08.283ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
55692025-05-08T22:07:08.283ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 19 reason = Timeout
55702025-05-08T22:07:08.283ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 18 reason = Timeout
55712025-05-08T22:07:08.283ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
55722025-05-08T22:07:08.283ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
55732025-05-08T22:07:08.283ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 18 reason = Timeout
55742025-05-08T22:07:08.283ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
55752025-05-08T22:07:08.283ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
55762025-05-08T22:07:08.283ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 18 reason = Timeout
55772025-05-08T22:07:08.283ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
55782025-05-08T22:07:08.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
55792025-05-08T22:07:08.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
55802025-05-08T22:07:08.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
55812025-05-08T22:07:08.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
55822025-05-08T22:07:08.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
55832025-05-08T22:07:08.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
55842025-05-08T22:07:08.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
55852025-05-08T22:07:08.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
55862025-05-08T22:07:08.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
55872025-05-08T22:07:08.284ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 18 reason = Timeout
55882025-05-08T22:07:08.284ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 18 reason = Timeout
55892025-05-08T22:07:08.284ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
55902025-05-08T22:07:08.284ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
55912025-05-08T22:07:08.284ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 18 reason = Timeout
55922025-05-08T22:07:08.284ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
55932025-05-08T22:07:08.284ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 18 reason = Timeout
55942025-05-08T22:07:08.284ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
55952025-05-08T22:07:08.284ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 18 reason = Timeout
55962025-05-08T22:07:08.284ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
55972025-05-08T22:07:08.284ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
55982025-05-08T22:07:08.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
55992025-05-08T22:07:08.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
56002025-05-08T22:07:08.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
56012025-05-08T22:07:08.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
56022025-05-08T22:07:08.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
56032025-05-08T22:07:08.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
56042025-05-08T22:07:08.285ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56052025-05-08T22:07:08.285ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56062025-05-08T22:07:08.288ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56072025-05-08T22:07:08.288ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56082025-05-08T22:07:08.289ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
56092025-05-08T22:07:08.289ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
56102025-05-08T22:07:08.289ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
56112025-05-08T22:07:08.291ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
56122025-05-08T22:07:08.291ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
56132025-05-08T22:07:08.291ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
56142025-05-08T22:07:08.291ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
56152025-05-08T22:07:08.291ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
56162025-05-08T22:07:08.291ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
56172025-05-08T22:07:08.291ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
56182025-05-08T22:07:08.291ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
56192025-05-08T22:07:08.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
56202025-05-08T22:07:08.309ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
56212025-05-08T22:07:08.309ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
56222025-05-08T22:07:08.310ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
56232025-05-08T22:07:08.310ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
56242025-05-08T22:07:08.315ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56252025-05-08T22:07:08.319ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 35.310637ms iteration = 18 status = {"errors":[],"requests_created_ok":[]}
56262025-05-08T22:07:08.320ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
56272025-05-08T22:07:08.320ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
56282025-05-08T22:07:08.320ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
56292025-05-08T22:07:08.334ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
56302025-05-08T22:07:08.334ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
56312025-05-08T22:07:08.336ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
56322025-05-08T22:07:08.336ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
56332025-05-08T22:07:08.336ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
56342025-05-08T22:07:08.337ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
56352025-05-08T22:07:08.337ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "88067363-f7e9-4db4-ab11-43d4a7d048c5", "content-length": "26", "date": "Thu, 08 May 2025 22:07:08 GMT"} })
56362025-05-08T22:07:08.337ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
56372025-05-08T22:07:08.339ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56382025-05-08T22:07:08.339ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
56392025-05-08T22:07:08.339ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 69.302941ms iteration = 52 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
56402025-05-08T22:07:08.339ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
56412025-05-08T22:07:08.341ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
56422025-05-08T22:07:08.341ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
56432025-05-08T22:07:08.341ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
56442025-05-08T22:07:08.342ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
56452025-05-08T22:07:08.343ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
56462025-05-08T22:07:08.343ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
56472025-05-08T22:07:08.343ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
56482025-05-08T22:07:08.349ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56492025-05-08T22:07:08.349ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
56502025-05-08T22:07:08.349ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
56512025-05-08T22:07:08.349ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
56522025-05-08T22:07:08.350ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
56532025-05-08T22:07:08.351ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
56542025-05-08T22:07:08.351ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
56552025-05-08T22:07:08.351ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
56562025-05-08T22:07:08.353ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 70.404313ms iteration = 52 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
56572025-05-08T22:07:08.354ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1404, tv_nsec: 94197406 } })
56582025-05-08T22:07:08.355ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
56592025-05-08T22:07:08.356ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
56602025-05-08T22:07:08.356ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
56612025-05-08T22:07:08.356ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ddff657-1e51-47e0-8668-08c68ce854c1", "content-length": "26", "date": "Thu, 08 May 2025 22:07:08 GMT"} })
56622025-05-08T22:07:08.356ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
56632025-05-08T22:07:08.358ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
56642025-05-08T22:07:08.360ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 75.886562ms iteration = 18 status = {"errors":[],"finish_invoked_ok":[]}
56652025-05-08T22:07:08.361ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 76.642429ms iteration = 18 status = {"errors":[],"garbage_collect_requested":[]}
56662025-05-08T22:07:08.362ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
56672025-05-08T22:07:08.363ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
56682025-05-08T22:07:08.364ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
56692025-05-08T22:07:08.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
56702025-05-08T22:07:08.365ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56712025-05-08T22:07:08.366ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56722025-05-08T22:07:08.367ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:552 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
56732025-05-08T22:07:08.367ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 83.919087ms iteration = 18 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
56742025-05-08T22:07:08.369ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
56752025-05-08T22:07:08.369ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
56762025-05-08T22:07:08.369ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
56772025-05-08T22:07:08.369ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
56782025-05-08T22:07:08.369ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
56792025-05-08T22:07:08.369ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
56802025-05-08T22:07:08.380ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:362
56812025-05-08T22:07:08.380ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 108.854344ms iteration = 18 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
56822025-05-08T22:07:08.388ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 105.393257ms iteration = 18 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
56832025-05-08T22:07:08.403ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
56842025-05-08T22:07:08.403ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
56852025-05-08T22:07:08.403ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 120.371147ms iteration = 18 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
56862025-05-08T22:07:08.410ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 125.810736ms iteration = 18 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
56872025-05-08T22:07:08.411ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
56882025-05-08T22:07:08.412ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
56892025-05-08T22:07:08.434ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
56902025-05-08T22:07:08.434ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
56912025-05-08T22:07:08.442ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 158.191482ms iteration = 18 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
56922025-05-08T22:07:08.446ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
56932025-05-08T22:07:08.447ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
56942025-05-08T22:07:08.453ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e zone_address = fd00:1122:3344:101::2
56952025-05-08T22:07:08.453ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
56962025-05-08T22:07:08.453ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e2e5dab-4e44-445c-9cff-49118452229f", "content-length": "26", "date": "Thu, 08 May 2025 22:07:08 GMT"} })
56972025-05-08T22:07:08.453ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1174 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
56982025-05-08T22:07:08.455ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
56992025-05-08T22:07:08.462ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 195.211038ms iteration = 18 status = {}
57002025-05-08T22:07:08.472ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 189.208243ms iteration = 19 status = {"physical_disks_added":0}
57012025-05-08T22:07:08.477ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
57022025-05-08T22:07:08.477ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
57032025-05-08T22:07:08.478ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "dbc4b382-fa45-4470-803c-0286742f40d4", "content-length": "372", "date": "Thu, 08 May 2025 22:07:08 GMT"} })
57042025-05-08T22:07:08.478ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
57052025-05-08T22:07:08.478ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
57062025-05-08T22:07:08.480ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 212.444012ms iteration = 18 status = null
57072025-05-08T22:07:08.485ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
57082025-05-08T22:07:08.486ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
57092025-05-08T22:07:08.487ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57102025-05-08T22:07:08.489ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
57112025-05-08T22:07:08.489ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
57122025-05-08T22:07:08.490ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
57132025-05-08T22:07:08.555ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57142025-05-08T22:07:08.556ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
57152025-05-08T22:07:08.556ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
57162025-05-08T22:07:08.556ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "14e43cab-988f-4eb7-8a99-f0d1546fe1a6", "date": "Thu, 08 May 2025 22:07:08 GMT"} })
57172025-05-08T22:07:08.556ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
57182025-05-08T22:07:08.557ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
57192025-05-08T22:07:08.558ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
57202025-05-08T22:07:08.558ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
57212025-05-08T22:07:08.558ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
57222025-05-08T22:07:08.559ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
57232025-05-08T22:07:08.559ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
57242025-05-08T22:07:08.560ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
57252025-05-08T22:07:08.560ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
57262025-05-08T22:07:08.562ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
57272025-05-08T22:07:08.562ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
57282025-05-08T22:07:08.564ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57292025-05-08T22:07:08.573ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 2f2f2c32-8d58-4801-a669-9618eac0c3f7, lookup_type: ById(2f2f2c32-8d58-4801-a669-9618eac0c3f7) } result = Ok(())
57302025-05-08T22:07:08.573ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
57312025-05-08T22:07:08.573ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
57322025-05-08T22:07:08.573ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
57332025-05-08T22:07:08.574ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
57342025-05-08T22:07:08.576ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 292.932663ms iteration = 18 status = 0
57352025-05-08T22:07:08.577ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
57362025-05-08T22:07:08.577ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
57372025-05-08T22:07:08.577ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "ea50bd11-cda0-43f1-8941-7f942eb35e3e", "date": "Thu, 08 May 2025 22:07:08 GMT"} })
57382025-05-08T22:07:08.579ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 296.508951ms iteration = 18 status = {}
57392025-05-08T22:07:08.587ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
57402025-05-08T22:07:08.588ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
57412025-05-08T22:07:08.590ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: cd0b930c-534b-46ad-ac3b-2eaa94b0bf2c, rack_id: 64f3d6f4-89b8-48aa-a622-b684e261667e, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(65f83a36-18fb-49b7-9914-87c8517d200e) } rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
57422025-05-08T22:07:08.590ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
57432025-05-08T22:07:08.590ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
57442025-05-08T22:07:08.602ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1833 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
57452025-05-08T22:07:08.602ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1860 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 65f83a36-18fb-49b7-9914-87c8517d200e
57462025-05-08T22:07:08.603ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
57472025-05-08T22:07:08.603ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
57482025-05-08T22:07:08.603ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
57492025-05-08T22:07:08.603ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
57502025-05-08T22:07:08.603ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
57512025-05-08T22:07:08.603ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2007 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
57522025-05-08T22:07:08.603ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
57532025-05-08T22:07:08.603ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2043 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
57542025-05-08T22:07:08.603ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
57552025-05-08T22:07:08.603ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
57562025-05-08T22:07:08.604ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
57572025-05-08T22:07:08.609ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
57582025-05-08T22:07:08.609ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
57592025-05-08T22:07:08.674ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
57602025-05-08T22:07:08.729ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "0599d4ef-6c05-43a1-bd11-01c2bbfd746a", "date": "Thu, 08 May 2025 22:07:08 GMT"} })
57612025-05-08T22:07:08.729ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch = Switch0
57622025-05-08T22:07:08.729ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
57632025-05-08T22:07:08.730ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "015441a4-4536-4dfa-9803-6e22553c66b1", "content-length": "682", "date": "Thu, 08 May 2025 22:07:08 GMT"} })
57642025-05-08T22:07:08.730ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
57652025-05-08T22:07:08.731ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
57662025-05-08T22:07:08.731ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "82bffe52-60b8-4d73-ac46-8be077f6e0b4", "content-length": "691", "date": "Thu, 08 May 2025 22:07:08 GMT"} })
57672025-05-08T22:07:08.731ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
57682025-05-08T22:07:08.731ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
57692025-05-08T22:07:08.731ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
57702025-05-08T22:07:08.733ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 450.299266ms iteration = 18 status = {}
57712025-05-08T22:07:08.734ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
57722025-05-08T22:07:08.735ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
57732025-05-08T22:07:08.738ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
57742025-05-08T22:07:08.738ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
57752025-05-08T22:07:08.740ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
57762025-05-08T22:07:08.740ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
57772025-05-08T22:07:08.748ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 464.70633ms iteration = 18 status = {}
57782025-05-08T22:07:09.192ZDEBGnexus (producer-registration-task): registering as oximeter producer with Nexus registration_count = 3
57792025-05-08T22:07:09.192ZDEBGnexus (producer-registration-task): will register with Nexus at [fd00:1122:3344:101::b]:12221
57802025-05-08T22:07:09.222ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::b]:12221/metrics/producers
57812025-05-08T22:07:09.222ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::b]:57430
57822025-05-08T22:07:09.223ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:57430 req_id = a50fbf10-f217-420b-97e1-d054dc1c750d roles = RoleSet { roles: {} } uri = /metrics/producers
57832025-05-08T22:07:09.223ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:57430 req_id = a50fbf10-f217-420b-97e1-d054dc1c750d resource = Database result = Ok(()) uri = /metrics/producers
57842025-05-08T22:07:09.263ZINFOnexus: assigned collector to new producer collector_id = 22ae79e2-3731-44c9-8346-78c5ddbfe577 file = nexus/src/app/oximeter.rs:76 producer_id = 85a39cf9-9419-458f-af2e-c8b40d8f5f79
57852025-05-08T22:07:09.263ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 40822 local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:57430 req_id = a50fbf10-f217-420b-97e1-d054dc1c750d response_code = 201 uri = /metrics/producers
57862025-05-08T22:07:09.263ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "a50fbf10-f217-420b-97e1-d054dc1c750d", "content-length": "41", "date": "Thu, 08 May 2025 22:07:09 GMT"} })
57872025-05-08T22:07:09.264ZDEBGnexus (producer-registration-task): successfully registered with Nexus, pausing until time to renew lease lease_duration = 600s wait_period = 150s
57882025-05-08T22:07:11.937ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 269 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 0fce1833-248f-4ceb-b33f-9ffac5e27a9e response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
57892025-05-08T22:07:18.269ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 53 reason = Timeout
57902025-05-08T22:07:18.269ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
57912025-05-08T22:07:18.270ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
57922025-05-08T22:07:18.282ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 53 reason = Timeout
57932025-05-08T22:07:18.282ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
57942025-05-08T22:07:18.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
57952025-05-08T22:07:18.302ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57962025-05-08T22:07:18.305ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
57972025-05-08T22:07:18.305ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
57982025-05-08T22:07:18.306ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
57992025-05-08T22:07:18.308ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
58002025-05-08T22:07:18.308ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
58012025-05-08T22:07:18.321ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
58022025-05-08T22:07:18.321ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
58032025-05-08T22:07:18.339ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58042025-05-08T22:07:18.342ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
58052025-05-08T22:07:18.342ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
58062025-05-08T22:07:18.342ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
58072025-05-08T22:07:18.352ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 69.280801ms iteration = 53 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
58082025-05-08T22:07:18.354ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
58092025-05-08T22:07:18.354ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 84.20824ms iteration = 53 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
58102025-05-08T22:07:21.938ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 286 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 5c941723-45b8-4b8d-abd5-b7a30d4578ee response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
58112025-05-08T22:07:28.269ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 54 reason = Timeout
58122025-05-08T22:07:28.269ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
58132025-05-08T22:07:28.269ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
58142025-05-08T22:07:28.271ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58152025-05-08T22:07:28.273ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
58162025-05-08T22:07:28.273ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
58172025-05-08T22:07:28.273ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
58182025-05-08T22:07:28.275ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
58192025-05-08T22:07:28.275ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
58202025-05-08T22:07:28.277ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58212025-05-08T22:07:28.280ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
58222025-05-08T22:07:28.280ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
58232025-05-08T22:07:28.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
58242025-05-08T22:07:28.282ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 54 reason = Timeout
58252025-05-08T22:07:28.282ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
58262025-05-08T22:07:28.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
58272025-05-08T22:07:28.293ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
58282025-05-08T22:07:28.293ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 23.670423ms iteration = 54 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
58292025-05-08T22:07:28.320ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
58302025-05-08T22:07:28.320ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
58312025-05-08T22:07:28.322ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 39.937267ms iteration = 54 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
58322025-05-08T22:07:31.938ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 280 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = f844cc3e-d195-4230-ab52-af322ffe7d87 response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
58332025-05-08T22:07:38.267ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 19 reason = Timeout
58342025-05-08T22:07:38.267ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
58352025-05-08T22:07:38.267ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 10 reason = Timeout
58362025-05-08T22:07:38.267ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 10 reason = Timeout
58372025-05-08T22:07:38.267ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-05-08 21:57:38.267561144 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
58382025-05-08T22:07:38.267ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 12 reason = Timeout
58392025-05-08T22:07:38.267ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
58402025-05-08T22:07:38.267ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 12 reason = Timeout
58412025-05-08T22:07:38.267ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 10 reason = Timeout
58422025-05-08T22:07:38.267ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 10 reason = Timeout
58432025-05-08T22:07:38.267ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
58442025-05-08T22:07:38.267ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 10 reason = Timeout
58452025-05-08T22:07:38.267ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
58462025-05-08T22:07:38.267ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
58472025-05-08T22:07:38.267ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 19 reason = Timeout
58482025-05-08T22:07:38.267ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
58492025-05-08T22:07:38.267ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 10 reason = Timeout
58502025-05-08T22:07:38.267ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
58512025-05-08T22:07:38.267ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
58522025-05-08T22:07:38.267ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
58532025-05-08T22:07:38.268ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
58542025-05-08T22:07:38.268ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
58552025-05-08T22:07:38.268ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
58562025-05-08T22:07:38.268ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
58572025-05-08T22:07:38.268ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("228ed1c1-3cc1-4055-aae9-5595ad856b86.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("8b245480-f91d-406b-bb08-850e1c3e3fc5.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c8f20f67-ea63-4dbc-b0f1-c30876daff82.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c8f20f67-ea63-4dbc-b0f1-c30876daff82.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 1434, tv_nsec: 7489199 } })
58582025-05-08T22:07:38.268ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
58592025-05-08T22:07:38.268ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("0ccabf41-81cd-45e7-bb26-335edaa2ee47.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("38a08dd6-6a75-4558-b4f8-df853f4d03d9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("38a08dd6-6a75-4558-b4f8-df853f4d03d9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::9))) }], valid_until: Instant { tv_sec: 1434, tv_nsec: 7612950 } })
58602025-05-08T22:07:38.268ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } dns_group = external
58612025-05-08T22:07:38.268ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.317774ms iteration = 10 status = {"addresses":["[fd00:1122:3344:101::8]:5353","[fd00:1122:3344:101::9]:5353"]}
58622025-05-08T22:07:38.269ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
58632025-05-08T22:07:38.269ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.366015ms iteration = 10 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:1::1]:5353"]}
58642025-05-08T22:07:38.269ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
58652025-05-08T22:07:38.269ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 55 reason = Timeout
58662025-05-08T22:07:38.269ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 11 reason = Timeout
58672025-05-08T22:07:38.269ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
58682025-05-08T22:07:38.269ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:85 reason = blueprint disabled target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2
58692025-05-08T22:07:38.269ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 19 reason = Timeout
58702025-05-08T22:07:38.269ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 13.2\u{b5}s iteration = 11 status = {"enabled":false,"target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2"}
58712025-05-08T22:07:38.270ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
58722025-05-08T22:07:38.270ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
58732025-05-08T22:07:38.270ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1434, tv_nsec: 9596031 } })
58742025-05-08T22:07:38.270ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58752025-05-08T22:07:38.270ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163
58762025-05-08T22:07:38.271ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
58772025-05-08T22:07:38.271ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
58782025-05-08T22:07:38.272ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 19 reason = Timeout
58792025-05-08T22:07:38.272ZDEBGnexus (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 5 reason = Timeout
58802025-05-08T22:07:38.272ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
58812025-05-08T22:07:38.272ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
58822025-05-08T22:07:38.272ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-05-08 21:57:38.267561144 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
58832025-05-08T22:07:38.272ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 4.539959ms iteration = 10 status = {"expiration":"2025-05-08T21:57:38.267561144Z","pruned":[]}
58842025-05-08T22:07:38.272ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1434, tv_nsec: 11543792 } })
58852025-05-08T22:07:38.272ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
58862025-05-08T22:07:38.272ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
58872025-05-08T22:07:38.272ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
58882025-05-08T22:07:38.272ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1163
58892025-05-08T22:07:38.272ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
58902025-05-08T22:07:38.272ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
58912025-05-08T22:07:38.272ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
58922025-05-08T22:07:38.272ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
58932025-05-08T22:07:38.272ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
58942025-05-08T22:07:38.272ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
58952025-05-08T22:07:38.272ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
58962025-05-08T22:07:38.273ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
58972025-05-08T22:07:38.273ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
58982025-05-08T22:07:38.273ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
58992025-05-08T22:07:38.275ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59002025-05-08T22:07:38.275ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59012025-05-08T22:07:38.276ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59022025-05-08T22:07:38.277ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 7.270068ms iteration = 19 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
59032025-05-08T22:07:38.279ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
59042025-05-08T22:07:38.279ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
59052025-05-08T22:07:38.279ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
59062025-05-08T22:07:38.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
59072025-05-08T22:07:38.280ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
59082025-05-08T22:07:38.280ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
59092025-05-08T22:07:38.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
59102025-05-08T22:07:38.282ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 10 reason = Timeout
59112025-05-08T22:07:38.282ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 55 reason = Timeout
59122025-05-08T22:07:38.282ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 10 reason = Timeout
59132025-05-08T22:07:38.282ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 20 reason = Timeout
59142025-05-08T22:07:38.282ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:53
59152025-05-08T22:07:38.282ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
59162025-05-08T22:07:38.282ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
59172025-05-08T22:07:38.282ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 19 reason = Timeout
59182025-05-08T22:07:38.282ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
59192025-05-08T22:07:38.282ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
59202025-05-08T22:07:38.282ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
59212025-05-08T22:07:38.282ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 19 reason = Timeout
59222025-05-08T22:07:38.282ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
59232025-05-08T22:07:38.282ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 19 reason = Timeout
59242025-05-08T22:07:38.282ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
59252025-05-08T22:07:38.282ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 19 reason = Timeout
59262025-05-08T22:07:38.282ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
59272025-05-08T22:07:38.282ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
59282025-05-08T22:07:38.282ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 19 reason = Timeout
59292025-05-08T22:07:38.282ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
59302025-05-08T22:07:38.282ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
59312025-05-08T22:07:38.282ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 19 reason = Timeout
59322025-05-08T22:07:38.282ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
59332025-05-08T22:07:38.282ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 19 reason = Timeout
59342025-05-08T22:07:38.282ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
59352025-05-08T22:07:38.282ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59362025-05-08T22:07:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
59372025-05-08T22:07:38.282ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
59382025-05-08T22:07:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
59392025-05-08T22:07:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
59402025-05-08T22:07:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
59412025-05-08T22:07:38.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
59422025-05-08T22:07:38.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
59432025-05-08T22:07:38.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
59442025-05-08T22:07:38.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
59452025-05-08T22:07:38.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
59462025-05-08T22:07:38.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
59472025-05-08T22:07:38.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
59482025-05-08T22:07:38.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
59492025-05-08T22:07:38.284ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
59502025-05-08T22:07:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
59512025-05-08T22:07:38.284ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59522025-05-08T22:07:38.284ZDEBGnexus (BackgroundTasks): activating background_task = webhook_deliverator iteration = 10 reason = Timeout
59532025-05-08T22:07:38.284ZDEBGnexus (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} }
59542025-05-08T22:07:38.284ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 10 reason = Timeout
59552025-05-08T22:07:38.284ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
59562025-05-08T22:07:38.284ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 19 reason = Timeout
59572025-05-08T22:07:38.284ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
59582025-05-08T22:07:38.284ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 19 reason = Timeout
59592025-05-08T22:07:38.284ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
59602025-05-08T22:07:38.285ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 19 reason = Timeout
59612025-05-08T22:07:38.285ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
59622025-05-08T22:07:38.285ZDEBGnexus (BackgroundTasks): activating background_task = webhook_dispatcher iteration = 10 reason = Timeout
59632025-05-08T22:07:38.285ZDEBGnexus (BackgroundTasks): roles background_task = webhook_dispatcher roles = RoleSet { roles: {} }
59642025-05-08T22:07:38.285ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 10 reason = Timeout
59652025-05-08T22:07:38.285ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
59662025-05-08T22:07:38.285ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 19 reason = Timeout
59672025-05-08T22:07:38.285ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
59682025-05-08T22:07:38.285ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 19 reason = Timeout
59692025-05-08T22:07:38.285ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
59702025-05-08T22:07:38.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
59712025-05-08T22:07:38.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
59722025-05-08T22:07:38.285ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
59732025-05-08T22:07:38.285ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
59742025-05-08T22:07:38.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
59752025-05-08T22:07:38.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database result = Ok(())
59762025-05-08T22:07:38.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
59772025-05-08T22:07:38.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database result = Ok(())
59782025-05-08T22:07:38.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
59792025-05-08T22:07:38.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
59802025-05-08T22:07:38.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
59812025-05-08T22:07:38.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
59822025-05-08T22:07:38.285ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59832025-05-08T22:07:38.287ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
59842025-05-08T22:07:38.287ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
59852025-05-08T22:07:38.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
59862025-05-08T22:07:38.287ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
59872025-05-08T22:07:38.287ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
59882025-05-08T22:07:38.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
59892025-05-08T22:07:38.291ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59902025-05-08T22:07:38.292ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59912025-05-08T22:07:38.296ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
59922025-05-08T22:07:38.296ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
59932025-05-08T22:07:38.296ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
59942025-05-08T22:07:38.296ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
59952025-05-08T22:07:38.297ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
59962025-05-08T22:07:38.297ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
59972025-05-08T22:07:38.297ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
59982025-05-08T22:07:38.297ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
59992025-05-08T22:07:38.299ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60002025-05-08T22:07:38.299ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60012025-05-08T22:07:38.302ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
60022025-05-08T22:07:38.302ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
60032025-05-08T22:07:38.303ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
60042025-05-08T22:07:38.303ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
60052025-05-08T22:07:38.303ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
60062025-05-08T22:07:38.304ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
60072025-05-08T22:07:38.304ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
60082025-05-08T22:07:38.308ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
60092025-05-08T22:07:38.309ZDEBGnexus (BackgroundTasks): activation complete background_task = webhook_deliverator elapsed = 24.772415ms iteration = 10 status = {"by_rx":{},"error":null}
60102025-05-08T22:07:38.314ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 29.031251ms iteration = 10 status = {"errors":[],"found_port_ok":[]}
60112025-05-08T22:07:38.322ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60122025-05-08T22:07:38.322ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60132025-05-08T22:07:38.325ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
60142025-05-08T22:07:38.325ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
60152025-05-08T22:07:38.326ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
60162025-05-08T22:07:38.326ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
60172025-05-08T22:07:38.329ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60182025-05-08T22:07:38.329ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
60192025-05-08T22:07:38.329ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
60202025-05-08T22:07:38.331ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
60212025-05-08T22:07:38.331ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
60222025-05-08T22:07:38.331ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
60232025-05-08T22:07:38.331ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
60242025-05-08T22:07:38.331ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
60252025-05-08T22:07:38.332ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
60262025-05-08T22:07:38.332ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
60272025-05-08T22:07:38.332ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
60282025-05-08T22:07:38.335ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
60292025-05-08T22:07:38.336ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
60302025-05-08T22:07:38.336ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60312025-05-08T22:07:38.337ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60322025-05-08T22:07:38.339ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60332025-05-08T22:07:38.340ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
60342025-05-08T22:07:38.340ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
60352025-05-08T22:07:38.341ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
60362025-05-08T22:07:38.341ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
60372025-05-08T22:07:38.341ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
60382025-05-08T22:07:38.342ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
60392025-05-08T22:07:38.343ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
60402025-05-08T22:07:38.343ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
60412025-05-08T22:07:38.343ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
60422025-05-08T22:07:38.351ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1434, tv_nsec: 90436586 } })
60432025-05-08T22:07:38.351ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
60442025-05-08T22:07:38.354ZDEBGnexus (BackgroundTasks): roles background_task = webhook_dispatcher roles = RoleSet { roles: {} }
60452025-05-08T22:07:38.354ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database result = Ok(())
60462025-05-08T22:07:38.357ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60472025-05-08T22:07:38.358ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60482025-05-08T22:07:38.361ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
60492025-05-08T22:07:38.362ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
60502025-05-08T22:07:38.362ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
60512025-05-08T22:07:38.362ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
60522025-05-08T22:07:38.362ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
60532025-05-08T22:07:38.365ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
60542025-05-08T22:07:38.365ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
60552025-05-08T22:07:38.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
60562025-05-08T22:07:38.366ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60572025-05-08T22:07:38.368ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
60582025-05-08T22:07:38.369ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
60592025-05-08T22:07:38.369ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
60602025-05-08T22:07:38.369ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
60612025-05-08T22:07:38.369ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
60622025-05-08T22:07:38.369ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
60632025-05-08T22:07:38.369ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
60642025-05-08T22:07:38.379ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60652025-05-08T22:07:38.379ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 94.806554ms iteration = 19 status = {"errors":[],"finish_invoked_ok":[]}
60662025-05-08T22:07:38.381ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
60672025-05-08T22:07:38.381ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 111.595534ms iteration = 55 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
60682025-05-08T22:07:38.383ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
60692025-05-08T22:07:38.383ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
60702025-05-08T22:07:38.383ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
60712025-05-08T22:07:38.385ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
60722025-05-08T22:07:38.385ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
60732025-05-08T22:07:38.385ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
60742025-05-08T22:07:38.385ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
60752025-05-08T22:07:38.385ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
60762025-05-08T22:07:38.386ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 101.652307ms iteration = 19 status = {"errors":[],"garbage_collect_requested":[]}
60772025-05-08T22:07:38.386ZDEBGnexus (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 114.864348ms iteration = 5 status = {"nsuccess":5}
60782025-05-08T22:07:38.388ZDEBGnexus (BackgroundTasks): activation complete background_task = webhook_dispatcher elapsed = 103.262324ms iteration = 10 status = {"dispatched":[],"errors":[],"glob_version":"140.0.0","globs_reprocessed":{},"no_receivers":[]}
60792025-05-08T22:07:38.389ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
60802025-05-08T22:07:38.389ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
60812025-05-08T22:07:38.397ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60822025-05-08T22:07:38.398ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "becf47f0-51b4-410f-bc50-d708fc32f9f9", "content-length": "26", "date": "Thu, 08 May 2025 22:07:38 GMT"} })
60832025-05-08T22:07:38.398ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
60842025-05-08T22:07:38.398ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
60852025-05-08T22:07:38.399ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
60862025-05-08T22:07:38.399ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
60872025-05-08T22:07:38.403ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
60882025-05-08T22:07:38.403ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
60892025-05-08T22:07:38.403ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:552 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
60902025-05-08T22:07:38.403ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 120.493468ms iteration = 19 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
60912025-05-08T22:07:38.403ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
60922025-05-08T22:07:38.405ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
60932025-05-08T22:07:38.405ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
60942025-05-08T22:07:38.405ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 120.463678ms iteration = 10 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
60952025-05-08T22:07:38.405ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60962025-05-08T22:07:38.406ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
60972025-05-08T22:07:38.406ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
60982025-05-08T22:07:38.407ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 122.789524ms iteration = 19 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
60992025-05-08T22:07:38.408ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 123.200578ms iteration = 19 status = {"errors":[],"requests_created_ok":[]}
61002025-05-08T22:07:38.408ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
61012025-05-08T22:07:38.408ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
61022025-05-08T22:07:38.408ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
61032025-05-08T22:07:38.409ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
61042025-05-08T22:07:38.409ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
61052025-05-08T22:07:38.422ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
61062025-05-08T22:07:38.422ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 140.070648ms iteration = 19 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
61072025-05-08T22:07:38.423ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
61082025-05-08T22:07:38.425ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
61092025-05-08T22:07:38.425ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
61102025-05-08T22:07:38.425ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
61112025-05-08T22:07:38.426ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
61122025-05-08T22:07:38.427ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61132025-05-08T22:07:38.428ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61142025-05-08T22:07:38.431ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
61152025-05-08T22:07:38.431ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
61162025-05-08T22:07:38.431ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
61172025-05-08T22:07:38.434ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 2f2f2c32-8d58-4801-a669-9618eac0c3f7, lookup_type: ById(2f2f2c32-8d58-4801-a669-9618eac0c3f7) } result = Ok(())
61182025-05-08T22:07:38.434ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
61192025-05-08T22:07:38.434ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
61202025-05-08T22:07:38.437ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
61212025-05-08T22:07:38.442ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 159.606837ms iteration = 55 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
61222025-05-08T22:07:38.443ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 160.897401ms iteration = 19 status = 0
61232025-05-08T22:07:38.443ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
61242025-05-08T22:07:38.443ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = d8128696-db5a-4b88-914a-69b28494e50e dns_zone_name = oxide.test found_so_far = 0 version = 2
61252025-05-08T22:07:38.443ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
61262025-05-08T22:07:38.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
61272025-05-08T22:07:38.447ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61282025-05-08T22:07:38.451ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
61292025-05-08T22:07:38.451ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:137 found = 0 not_ready_to_be_deleted = 0
61302025-05-08T22:07:38.451ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 169.093268ms iteration = 10 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
61312025-05-08T22:07:38.453ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
61322025-05-08T22:07:38.453ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
61332025-05-08T22:07:38.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
61342025-05-08T22:07:38.455ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
61352025-05-08T22:07:38.455ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
61362025-05-08T22:07:38.455ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
61372025-05-08T22:07:38.455ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
61382025-05-08T22:07:38.455ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 7586992a-6fc5-4073-8962-728778f16aa5 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
61392025-05-08T22:07:38.455ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
61402025-05-08T22:07:38.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
61412025-05-08T22:07:38.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
61422025-05-08T22:07:38.456ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = d8128696-db5a-4b88-914a-69b28494e50e dns_zone_name = oxide.test found_so_far = 1 version = 2
61432025-05-08T22:07:38.456ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
61442025-05-08T22:07:38.456ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-05-08 21:59:21.759368 UTC dns_group = external generation = 2
61452025-05-08T22:07:38.456ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 188.665858ms iteration = 10 status = {"generation":2}
61462025-05-08T22:07:38.458ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61472025-05-08T22:07:38.459ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-05-08T21:58:35.560911Z, time_modified: 2025-05-08T21:58:35.560911Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, db_silo: Silo { identity: SiloIdentity { id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-05-08T21:59:21.709488Z, time_modified: 2025-05-08T21:59:21.709488Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, db_silo: Silo { identity: SiloIdentity { id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-05-08T21:59:21.709488Z, time_modified: 2025-05-08T21:59:21.709488Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47" }] }) }
61482025-05-08T22:07:38.459ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 191.439168ms iteration = 10 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"7137ee58-6aa7-4199-a248-feadf3b8bafb","tls_certs":["37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47"]}},"default_endpoint":{"silo_id":"7137ee58-6aa7-4199-a248-feadf3b8bafb","tls_certs":["37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
61492025-05-08T22:07:38.461ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
61502025-05-08T22:07:38.461ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
61512025-05-08T22:07:38.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
61522025-05-08T22:07:38.465ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
61532025-05-08T22:07:38.465ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
61542025-05-08T22:07:38.465ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
61552025-05-08T22:07:38.465ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "e077ad10-8d7b-4a14-ab15-ffc34a408344", "content-length": "26", "date": "Thu, 08 May 2025 22:07:38 GMT"} })
61562025-05-08T22:07:38.465ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
61572025-05-08T22:07:38.466ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 181.664543ms iteration = 19 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
61582025-05-08T22:07:38.466ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 7586992a-6fc5-4073-8962-728778f16aa5 dns_zone_name = control-plane.oxide.internal found_so_far = 55 version = 1
61592025-05-08T22:07:38.466ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
61602025-05-08T22:07:38.466ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-05-08 21:59:21.182334 UTC dns_group = internal generation = 1
61612025-05-08T22:07:38.467ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 199.346271ms iteration = 10 status = {"generation":1}
61622025-05-08T22:07:38.467ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
61632025-05-08T22:07:38.488ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:362
61642025-05-08T22:07:38.488ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 216.037791ms iteration = 19 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
61652025-05-08T22:07:38.493ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
61662025-05-08T22:07:38.493ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
61672025-05-08T22:07:38.495ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
61682025-05-08T22:07:38.495ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
61692025-05-08T22:07:38.505ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
61702025-05-08T22:07:38.505ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
61712025-05-08T22:07:38.505ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 222.585ms iteration = 19 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
61722025-05-08T22:07:38.526ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e zone_address = fd00:1122:3344:101::2
61732025-05-08T22:07:38.526ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
61742025-05-08T22:07:38.527ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "d0589305-10b2-4a6e-b603-aeefbe61d0c8", "content-length": "26", "date": "Thu, 08 May 2025 22:07:38 GMT"} })
61752025-05-08T22:07:38.527ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1174 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
61762025-05-08T22:07:38.529ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
61772025-05-08T22:07:38.541ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 259.452935ms iteration = 20 status = {"physical_disks_added":0}
61782025-05-08T22:07:38.615ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 348.195524ms iteration = 19 status = {}
61792025-05-08T22:07:38.644ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
61802025-05-08T22:07:38.687ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
61812025-05-08T22:07:38.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
61822025-05-08T22:07:38.690ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61832025-05-08T22:07:38.694ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
61842025-05-08T22:07:38.694ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
61852025-05-08T22:07:38.694ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
61862025-05-08T22:07:38.696ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61872025-05-08T22:07:38.699ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
61882025-05-08T22:07:38.699ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
61892025-05-08T22:07:38.700ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
61902025-05-08T22:07:38.701ZINFOnexus (BackgroundTasks): instance reincarnation completed successfully background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:113 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0
61912025-05-08T22:07:38.702ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 419.499377ms iteration = 10 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
61922025-05-08T22:07:38.717ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
61932025-05-08T22:07:38.718ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
61942025-05-08T22:07:38.721ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
61952025-05-08T22:07:38.721ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
61962025-05-08T22:07:38.722ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "77036887-5240-4a46-baef-37de11d57eb6", "content-length": "372", "date": "Thu, 08 May 2025 22:07:38 GMT"} })
61972025-05-08T22:07:38.722ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
61982025-05-08T22:07:38.722ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
61992025-05-08T22:07:38.725ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 457.778196ms iteration = 19 status = null
62002025-05-08T22:07:38.731ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
62012025-05-08T22:07:38.731ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "c653ee99-fc0a-458d-b6e2-db68c7588b6b", "date": "Thu, 08 May 2025 22:07:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
62022025-05-08T22:07:38.731ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
62032025-05-08T22:07:38.737ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "efb3cb5e-f1ad-468c-9e29-20e6a6f9e5a1", "date": "Thu, 08 May 2025 22:07:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
62042025-05-08T22:07:38.737ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
62052025-05-08T22:07:38.739ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
62062025-05-08T22:07:38.739ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 471.428612ms iteration = 12 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
62072025-05-08T22:07:38.746ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
62082025-05-08T22:07:38.746ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
62092025-05-08T22:07:38.746ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "d0b8d9ab-505d-4927-b8ef-8e6c56bf93b2", "date": "Thu, 08 May 2025 22:07:38 GMT"} })
62102025-05-08T22:07:38.746ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
62112025-05-08T22:07:38.746ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
62122025-05-08T22:07:38.749ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
62132025-05-08T22:07:38.750ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
62142025-05-08T22:07:38.753ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
62152025-05-08T22:07:38.754ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
62162025-05-08T22:07:38.757ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
62172025-05-08T22:07:38.757ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
62182025-05-08T22:07:38.758ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "076e6735-30b6-4c2a-aeed-d8fbf67976b9", "date": "Thu, 08 May 2025 22:07:38 GMT"} })
62192025-05-08T22:07:38.758ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
62202025-05-08T22:07:38.759ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "80877da8-c06b-43c4-9eae-608ef2b07928", "date": "Thu, 08 May 2025 22:07:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
62212025-05-08T22:07:38.759ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
62222025-05-08T22:07:38.760ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 477.691229ms iteration = 19 status = {}
62232025-05-08T22:07:38.761ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "33722765-58eb-4fae-b80b-327158fae2ce", "date": "Thu, 08 May 2025 22:07:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
62242025-05-08T22:07:38.761ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
62252025-05-08T22:07:38.762ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "6caf52f6-b867-4145-97c7-ea694ebbabfb", "date": "Thu, 08 May 2025 22:07:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
62262025-05-08T22:07:38.762ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
62272025-05-08T22:07:38.764ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
62282025-05-08T22:07:38.764ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 497.044396ms iteration = 12 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
62292025-05-08T22:07:38.822ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
62302025-05-08T22:07:38.822ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
62312025-05-08T22:07:38.824ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: cd0b930c-534b-46ad-ac3b-2eaa94b0bf2c, rack_id: 64f3d6f4-89b8-48aa-a622-b684e261667e, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(65f83a36-18fb-49b7-9914-87c8517d200e) } rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
62322025-05-08T22:07:38.824ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
62332025-05-08T22:07:38.825ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
62342025-05-08T22:07:38.836ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1833 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
62352025-05-08T22:07:38.836ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1860 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 65f83a36-18fb-49b7-9914-87c8517d200e
62362025-05-08T22:07:38.836ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
62372025-05-08T22:07:38.836ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
62382025-05-08T22:07:38.836ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
62392025-05-08T22:07:38.836ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
62402025-05-08T22:07:38.836ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
62412025-05-08T22:07:38.836ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2007 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
62422025-05-08T22:07:38.836ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
62432025-05-08T22:07:38.836ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2043 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
62442025-05-08T22:07:38.836ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
62452025-05-08T22:07:38.836ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
62462025-05-08T22:07:38.836ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
62472025-05-08T22:07:38.838ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
62482025-05-08T22:07:38.838ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
62492025-05-08T22:07:38.838ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
62502025-05-08T22:07:38.929ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "c132520b-7ebf-4dbb-b8f5-7e5773011fe5", "content-length": "691", "date": "Thu, 08 May 2025 22:07:38 GMT"} })
62512025-05-08T22:07:38.929ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
62522025-05-08T22:07:38.929ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
62532025-05-08T22:07:38.929ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
62542025-05-08T22:07:38.931ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 649.331865ms iteration = 19 status = {}
62552025-05-08T22:07:39.034ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "858eace4-8218-45c6-b4f8-eb607b8995c3", "date": "Thu, 08 May 2025 22:07:38 GMT"} })
62562025-05-08T22:07:39.034ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch = Switch0
62572025-05-08T22:07:39.034ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
62582025-05-08T22:07:39.039ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "9fbb3a74-dc8e-42ed-8b74-dc6c7c9d0ad4", "content-length": "682", "date": "Thu, 08 May 2025 22:07:38 GMT"} })
62592025-05-08T22:07:39.040ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
62602025-05-08T22:07:39.040ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
62612025-05-08T22:07:39.044ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
62622025-05-08T22:07:39.044ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
62632025-05-08T22:07:39.048ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
62642025-05-08T22:07:39.048ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
62652025-05-08T22:07:39.050ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
62662025-05-08T22:07:39.050ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
62672025-05-08T22:07:39.057ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 774.700677ms iteration = 19 status = {}
62682025-05-08T22:07:41.937ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 259 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = cd88ac21-8520-47d4-8c5e-96b37c6d80cc response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
62692025-05-08T22:07:48.270ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 56 reason = Timeout
62702025-05-08T22:07:48.270ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
62712025-05-08T22:07:48.270ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
62722025-05-08T22:07:48.272ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62732025-05-08T22:07:48.275ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
62742025-05-08T22:07:48.275ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
62752025-05-08T22:07:48.275ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
62762025-05-08T22:07:48.277ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
62772025-05-08T22:07:48.277ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
62782025-05-08T22:07:48.279ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62792025-05-08T22:07:48.281ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
62802025-05-08T22:07:48.281ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
62812025-05-08T22:07:48.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
62822025-05-08T22:07:48.283ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 56 reason = Timeout
62832025-05-08T22:07:48.283ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
62842025-05-08T22:07:48.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
62852025-05-08T22:07:48.294ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
62862025-05-08T22:07:48.294ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 24.601063ms iteration = 56 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
62872025-05-08T22:07:48.320ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
62882025-05-08T22:07:48.321ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
62892025-05-08T22:07:48.353ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 70.069779ms iteration = 56 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
62902025-05-08T22:07:51.937ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 283 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 4a2e90f9-56ed-42c9-bbc3-66bf6fcae88d response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
62912025-05-08T22:07:58.270ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 57 reason = Timeout
62922025-05-08T22:07:58.270ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
62932025-05-08T22:07:58.270ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
62942025-05-08T22:07:58.272ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62952025-05-08T22:07:58.275ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
62962025-05-08T22:07:58.275ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
62972025-05-08T22:07:58.275ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
62982025-05-08T22:07:58.277ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
62992025-05-08T22:07:58.277ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
63002025-05-08T22:07:58.278ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63012025-05-08T22:07:58.281ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
63022025-05-08T22:07:58.281ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
63032025-05-08T22:07:58.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
63042025-05-08T22:07:58.282ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 57 reason = Timeout
63052025-05-08T22:07:58.282ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
63062025-05-08T22:07:58.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
63072025-05-08T22:07:58.285ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
63082025-05-08T22:07:58.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
63092025-05-08T22:07:58.286ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.519808ms iteration = 57 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
63102025-05-08T22:07:58.293ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
63112025-05-08T22:07:58.293ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 23.080696ms iteration = 57 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
63122025-05-08T22:08:01.937ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 277 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 2b235df6-74c1-4952-9b18-67fa8d8101a6 response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
63132025-05-08T22:08:08.268ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 20 reason = Timeout
63142025-05-08T22:08:08.268ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 20 reason = Timeout
63152025-05-08T22:08:08.268ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
63162025-05-08T22:08:08.268ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
63172025-05-08T22:08:08.268ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
63182025-05-08T22:08:08.268ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
63192025-05-08T22:08:08.270ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 58 reason = Timeout
63202025-05-08T22:08:08.270ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 20 reason = Timeout
63212025-05-08T22:08:08.270ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
63222025-05-08T22:08:08.270ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
63232025-05-08T22:08:08.270ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1464, tv_nsec: 9913865 } })
63242025-05-08T22:08:08.271ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1464, tv_nsec: 10183978 } })
63252025-05-08T22:08:08.271ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163
63262025-05-08T22:08:08.271ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1163
63272025-05-08T22:08:08.272ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 20 reason = Timeout
63282025-05-08T22:08:08.272ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
63292025-05-08T22:08:08.272ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
63302025-05-08T22:08:08.272ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63312025-05-08T22:08:08.273ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 2.962022ms iteration = 20 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
63322025-05-08T22:08:08.274ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63332025-05-08T22:08:08.275ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
63342025-05-08T22:08:08.275ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
63352025-05-08T22:08:08.275ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
63362025-05-08T22:08:08.276ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
63372025-05-08T22:08:08.276ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
63382025-05-08T22:08:08.276ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
63392025-05-08T22:08:08.277ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
63402025-05-08T22:08:08.278ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
63412025-05-08T22:08:08.279ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63422025-05-08T22:08:08.282ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 58 reason = Timeout
63432025-05-08T22:08:08.283ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
63442025-05-08T22:08:08.283ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 20 reason = Timeout
63452025-05-08T22:08:08.283ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
63462025-05-08T22:08:08.283ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
63472025-05-08T22:08:08.283ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 20 reason = Timeout
63482025-05-08T22:08:08.283ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
63492025-05-08T22:08:08.283ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
63502025-05-08T22:08:08.283ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 20 reason = Timeout
63512025-05-08T22:08:08.283ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
63522025-05-08T22:08:08.283ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 20 reason = Timeout
63532025-05-08T22:08:08.283ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
63542025-05-08T22:08:08.283ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 20 reason = Timeout
63552025-05-08T22:08:08.283ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 21 reason = Timeout
63562025-05-08T22:08:08.283ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
63572025-05-08T22:08:08.283ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
63582025-05-08T22:08:08.283ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
63592025-05-08T22:08:08.283ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
63602025-05-08T22:08:08.283ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 20 reason = Timeout
63612025-05-08T22:08:08.283ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
63622025-05-08T22:08:08.283ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 20 reason = Timeout
63632025-05-08T22:08:08.283ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
63642025-05-08T22:08:08.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
63652025-05-08T22:08:08.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
63662025-05-08T22:08:08.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
63672025-05-08T22:08:08.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
63682025-05-08T22:08:08.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
63692025-05-08T22:08:08.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
63702025-05-08T22:08:08.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
63712025-05-08T22:08:08.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
63722025-05-08T22:08:08.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
63732025-05-08T22:08:08.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
63742025-05-08T22:08:08.285ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 20 reason = Timeout
63752025-05-08T22:08:08.285ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 20 reason = Timeout
63762025-05-08T22:08:08.285ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
63772025-05-08T22:08:08.285ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
63782025-05-08T22:08:08.285ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 20 reason = Timeout
63792025-05-08T22:08:08.285ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
63802025-05-08T22:08:08.285ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 20 reason = Timeout
63812025-05-08T22:08:08.285ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
63822025-05-08T22:08:08.285ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 20 reason = Timeout
63832025-05-08T22:08:08.285ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
63842025-05-08T22:08:08.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
63852025-05-08T22:08:08.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
63862025-05-08T22:08:08.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
63872025-05-08T22:08:08.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
63882025-05-08T22:08:08.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
63892025-05-08T22:08:08.285ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63902025-05-08T22:08:08.288ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
63912025-05-08T22:08:08.288ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
63922025-05-08T22:08:08.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
63932025-05-08T22:08:08.288ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63942025-05-08T22:08:08.290ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63952025-05-08T22:08:08.290ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63962025-05-08T22:08:08.291ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
63972025-05-08T22:08:08.291ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
63982025-05-08T22:08:08.291ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
63992025-05-08T22:08:08.293ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
64002025-05-08T22:08:08.293ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
64012025-05-08T22:08:08.293ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
64022025-05-08T22:08:08.293ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
64032025-05-08T22:08:08.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
64042025-05-08T22:08:08.294ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
64052025-05-08T22:08:08.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
64062025-05-08T22:08:08.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
64072025-05-08T22:08:08.305ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
64082025-05-08T22:08:08.306ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 35.573861ms iteration = 58 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
64092025-05-08T22:08:08.307ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1464, tv_nsec: 46185223 } })
64102025-05-08T22:08:08.307ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
64112025-05-08T22:08:08.309ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
64122025-05-08T22:08:08.310ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
64132025-05-08T22:08:08.312ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64142025-05-08T22:08:08.319ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
64152025-05-08T22:08:08.319ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
64162025-05-08T22:08:08.319ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
64172025-05-08T22:08:08.344ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
64182025-05-08T22:08:08.344ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
64192025-05-08T22:08:08.347ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64202025-05-08T22:08:08.347ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
64212025-05-08T22:08:08.347ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
64222025-05-08T22:08:08.349ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
64232025-05-08T22:08:08.350ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
64242025-05-08T22:08:08.350ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
64252025-05-08T22:08:08.350ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
64262025-05-08T22:08:08.350ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
64272025-05-08T22:08:08.351ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 66.498152ms iteration = 20 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
64282025-05-08T22:08:08.353ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 68.588704ms iteration = 20 status = {"errors":[],"finish_invoked_ok":[]}
64292025-05-08T22:08:08.358ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
64302025-05-08T22:08:08.358ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
64312025-05-08T22:08:08.361ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64322025-05-08T22:08:08.362ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 76.986063ms iteration = 20 status = {"errors":[],"garbage_collect_requested":[]}
64332025-05-08T22:08:08.363ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
64342025-05-08T22:08:08.363ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
64352025-05-08T22:08:08.363ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
64362025-05-08T22:08:08.364ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
64372025-05-08T22:08:08.364ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
64382025-05-08T22:08:08.364ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
64392025-05-08T22:08:08.365ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c8ccc73-6532-446b-a835-10753442c045", "content-length": "26", "date": "Thu, 08 May 2025 22:08:08 GMT"} })
64402025-05-08T22:08:08.365ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
64412025-05-08T22:08:08.366ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
64422025-05-08T22:08:08.366ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
64432025-05-08T22:08:08.366ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
64442025-05-08T22:08:08.367ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
64452025-05-08T22:08:08.367ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 82.29012ms iteration = 20 status = {"errors":[],"requests_created_ok":[]}
64462025-05-08T22:08:08.367ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
64472025-05-08T22:08:08.367ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
64482025-05-08T22:08:08.369ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 84.596304ms iteration = 20 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
64492025-05-08T22:08:08.370ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 87.266984ms iteration = 58 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
64502025-05-08T22:08:08.370ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
64512025-05-08T22:08:08.370ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
64522025-05-08T22:08:08.370ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
64532025-05-08T22:08:08.370ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
64542025-05-08T22:08:08.371ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64552025-05-08T22:08:08.373ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
64562025-05-08T22:08:08.373ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:552 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
64572025-05-08T22:08:08.373ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
64582025-05-08T22:08:08.373ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 90.325666ms iteration = 20 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
64592025-05-08T22:08:08.373ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
64602025-05-08T22:08:08.373ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
64612025-05-08T22:08:08.373ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
64622025-05-08T22:08:08.376ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 93.229727ms iteration = 20 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
64632025-05-08T22:08:08.380ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
64642025-05-08T22:08:08.380ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
64652025-05-08T22:08:08.380ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "f250b1c1-797a-46f7-bff4-9fd0be60f113", "content-length": "26", "date": "Thu, 08 May 2025 22:08:08 GMT"} })
64662025-05-08T22:08:08.381ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
64672025-05-08T22:08:08.383ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
64682025-05-08T22:08:08.394ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
64692025-05-08T22:08:08.394ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
64702025-05-08T22:08:08.401ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
64712025-05-08T22:08:08.401ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
64722025-05-08T22:08:08.402ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:362
64732025-05-08T22:08:08.402ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 130.214352ms iteration = 20 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
64742025-05-08T22:08:08.403ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64752025-05-08T22:08:08.406ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 2f2f2c32-8d58-4801-a669-9618eac0c3f7, lookup_type: ById(2f2f2c32-8d58-4801-a669-9618eac0c3f7) } result = Ok(())
64762025-05-08T22:08:08.406ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
64772025-05-08T22:08:08.406ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
64782025-05-08T22:08:08.408ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 124.932406ms iteration = 20 status = 0
64792025-05-08T22:08:08.431ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
64802025-05-08T22:08:08.431ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
64812025-05-08T22:08:08.431ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 147.933843ms iteration = 20 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
64822025-05-08T22:08:08.442ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
64832025-05-08T22:08:08.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
64842025-05-08T22:08:08.454ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 171.10161ms iteration = 21 status = {"physical_disks_added":0}
64852025-05-08T22:08:08.456ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e zone_address = fd00:1122:3344:101::2
64862025-05-08T22:08:08.456ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
64872025-05-08T22:08:08.457ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "682569a1-0ed3-45d1-942e-ea9ee0e25977", "content-length": "26", "date": "Thu, 08 May 2025 22:08:08 GMT"} })
64882025-05-08T22:08:08.457ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1174 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
64892025-05-08T22:08:08.458ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
64902025-05-08T22:08:08.487ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
64912025-05-08T22:08:08.487ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
64922025-05-08T22:08:08.487ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab147c69-cbb4-4c6c-ba79-c1887023f95f", "content-length": "372", "date": "Thu, 08 May 2025 22:08:08 GMT"} })
64932025-05-08T22:08:08.487ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
64942025-05-08T22:08:08.488ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
64952025-05-08T22:08:08.499ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 231.412485ms iteration = 20 status = {}
64962025-05-08T22:08:08.503ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
64972025-05-08T22:08:08.503ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
64982025-05-08T22:08:08.504ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "9ff78a4b-184c-4bf5-8aea-6d4bd2d2dd51", "date": "Thu, 08 May 2025 22:08:08 GMT"} })
64992025-05-08T22:08:08.504ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
65002025-05-08T22:08:08.504ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
65012025-05-08T22:08:08.506ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
65022025-05-08T22:08:08.506ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
65032025-05-08T22:08:08.509ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
65042025-05-08T22:08:08.510ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
65052025-05-08T22:08:08.513ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
65062025-05-08T22:08:08.513ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
65072025-05-08T22:08:08.514ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "3faf8c8a-cad9-42a8-b26c-daca1567d0e9", "date": "Thu, 08 May 2025 22:08:08 GMT"} })
65082025-05-08T22:08:08.516ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 232.976282ms iteration = 20 status = {}
65092025-05-08T22:08:08.556ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
65102025-05-08T22:08:08.557ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
65112025-05-08T22:08:08.559ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 290.7566ms iteration = 20 status = null
65122025-05-08T22:08:08.619ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
65132025-05-08T22:08:08.619ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
65142025-05-08T22:08:08.620ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65152025-05-08T22:08:08.622ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
65162025-05-08T22:08:08.622ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
65172025-05-08T22:08:08.623ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
65182025-05-08T22:08:08.624ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65192025-05-08T22:08:08.626ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
65202025-05-08T22:08:08.626ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
65212025-05-08T22:08:08.626ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
65222025-05-08T22:08:08.638ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
65232025-05-08T22:08:08.639ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
65242025-05-08T22:08:08.640ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: cd0b930c-534b-46ad-ac3b-2eaa94b0bf2c, rack_id: 64f3d6f4-89b8-48aa-a622-b684e261667e, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(65f83a36-18fb-49b7-9914-87c8517d200e) } rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
65252025-05-08T22:08:08.640ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
65262025-05-08T22:08:08.641ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
65272025-05-08T22:08:08.662ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
65282025-05-08T22:08:08.673ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1833 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
65292025-05-08T22:08:08.673ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1860 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 65f83a36-18fb-49b7-9914-87c8517d200e
65302025-05-08T22:08:08.673ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
65312025-05-08T22:08:08.673ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
65322025-05-08T22:08:08.673ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
65332025-05-08T22:08:08.673ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
65342025-05-08T22:08:08.673ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
65352025-05-08T22:08:08.673ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2007 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
65362025-05-08T22:08:08.673ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
65372025-05-08T22:08:08.673ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2043 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
65382025-05-08T22:08:08.673ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
65392025-05-08T22:08:08.673ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
65402025-05-08T22:08:08.674ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
65412025-05-08T22:08:08.676ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
65422025-05-08T22:08:08.676ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
65432025-05-08T22:08:08.718ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "fcacb6c1-74c9-43f1-a964-105e94b7ad40", "content-length": "691", "date": "Thu, 08 May 2025 22:08:08 GMT"} })
65442025-05-08T22:08:08.718ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
65452025-05-08T22:08:08.718ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
65462025-05-08T22:08:08.718ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
65472025-05-08T22:08:08.719ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 436.094014ms iteration = 20 status = {}
65482025-05-08T22:08:08.815ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "c85b3e8f-bfa1-43a3-97f2-f47f183fa1a7", "date": "Thu, 08 May 2025 22:08:08 GMT"} })
65492025-05-08T22:08:08.815ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch = Switch0
65502025-05-08T22:08:08.815ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
65512025-05-08T22:08:08.815ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "9fa929c9-a90b-4ca8-8584-5155ff6150cf", "content-length": "682", "date": "Thu, 08 May 2025 22:08:08 GMT"} })
65522025-05-08T22:08:08.815ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
65532025-05-08T22:08:08.815ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
65542025-05-08T22:08:08.819ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
65552025-05-08T22:08:08.820ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
65562025-05-08T22:08:08.823ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
65572025-05-08T22:08:08.823ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
65582025-05-08T22:08:08.825ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
65592025-05-08T22:08:08.825ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
65602025-05-08T22:08:08.834ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 550.842872ms iteration = 20 status = {}
65612025-05-08T22:08:11.938ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 263 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = c8314c2e-acf6-4fee-a65e-7b4ae4d4c454 response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
65622025-05-08T22:08:18.270ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 59 reason = Timeout
65632025-05-08T22:08:18.270ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
65642025-05-08T22:08:18.271ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
65652025-05-08T22:08:18.283ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 59 reason = Timeout
65662025-05-08T22:08:18.283ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
65672025-05-08T22:08:18.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
65682025-05-08T22:08:18.285ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
65692025-05-08T22:08:18.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
65702025-05-08T22:08:18.306ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65712025-05-08T22:08:18.309ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
65722025-05-08T22:08:18.309ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
65732025-05-08T22:08:18.309ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
65742025-05-08T22:08:18.311ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
65752025-05-08T22:08:18.311ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
65762025-05-08T22:08:18.313ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65772025-05-08T22:08:18.316ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
65782025-05-08T22:08:18.316ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
65792025-05-08T22:08:18.316ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
65802025-05-08T22:08:18.322ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 38.732045ms iteration = 59 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
65812025-05-08T22:08:18.326ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
65822025-05-08T22:08:18.326ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 55.771767ms iteration = 59 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
65832025-05-08T22:08:21.938ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 244 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 0619f06f-da1f-462c-850f-fa93c855228a response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
65842025-05-08T22:08:28.270ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 60 reason = Timeout
65852025-05-08T22:08:28.271ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
65862025-05-08T22:08:28.271ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
65872025-05-08T22:08:28.283ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 60 reason = Timeout
65882025-05-08T22:08:28.283ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
65892025-05-08T22:08:28.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
65902025-05-08T22:08:28.285ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
65912025-05-08T22:08:28.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
65922025-05-08T22:08:28.314ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65932025-05-08T22:08:28.317ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
65942025-05-08T22:08:28.317ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
65952025-05-08T22:08:28.317ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
65962025-05-08T22:08:28.319ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
65972025-05-08T22:08:28.320ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
65982025-05-08T22:08:28.321ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65992025-05-08T22:08:28.324ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
66002025-05-08T22:08:28.324ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
66012025-05-08T22:08:28.324ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
66022025-05-08T22:08:28.326ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 42.801617ms iteration = 60 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
66032025-05-08T22:08:28.335ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
66042025-05-08T22:08:28.336ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 65.070056ms iteration = 60 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
66052025-05-08T22:08:31.938ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 250 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 3901bbd9-b00e-4d4b-b05b-69a797a5e6e2 response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
66062025-05-08T22:08:38.268ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 21 reason = Timeout
66072025-05-08T22:08:38.268ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
66082025-05-08T22:08:38.268ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 13 reason = Timeout
66092025-05-08T22:08:38.268ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 11 reason = Timeout
66102025-05-08T22:08:38.268ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 11 reason = Timeout
66112025-05-08T22:08:38.268ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
66122025-05-08T22:08:38.268ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 13 reason = Timeout
66132025-05-08T22:08:38.268ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 11 reason = Timeout
66142025-05-08T22:08:38.268ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
66152025-05-08T22:08:38.268ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 21 reason = Timeout
66162025-05-08T22:08:38.268ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
66172025-05-08T22:08:38.268ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
66182025-05-08T22:08:38.268ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 11 reason = Timeout
66192025-05-08T22:08:38.268ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 11 reason = Timeout
66202025-05-08T22:08:38.268ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
66212025-05-08T22:08:38.268ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 11 reason = Timeout
66222025-05-08T22:08:38.268ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
66232025-05-08T22:08:38.268ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-05-08 21:58:38.268898552 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
66242025-05-08T22:08:38.268ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
66252025-05-08T22:08:38.269ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
66262025-05-08T22:08:38.269ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
66272025-05-08T22:08:38.269ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
66282025-05-08T22:08:38.269ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
66292025-05-08T22:08:38.269ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
66302025-05-08T22:08:38.269ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
66312025-05-08T22:08:38.269ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("0ccabf41-81cd-45e7-bb26-335edaa2ee47.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("38a08dd6-6a75-4558-b4f8-df853f4d03d9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("38a08dd6-6a75-4558-b4f8-df853f4d03d9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::9))) }], valid_until: Instant { tv_sec: 1494, tv_nsec: 8107025 } })
66322025-05-08T22:08:38.269ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("228ed1c1-3cc1-4055-aae9-5595ad856b86.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("8b245480-f91d-406b-bb08-850e1c3e3fc5.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c8f20f67-ea63-4dbc-b0f1-c30876daff82.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c8f20f67-ea63-4dbc-b0f1-c30876daff82.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 1494, tv_nsec: 8112276 } })
66332025-05-08T22:08:38.270ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 61 reason = Timeout
66342025-05-08T22:08:38.270ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 12 reason = Timeout
66352025-05-08T22:08:38.270ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
66362025-05-08T22:08:38.270ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:85 reason = blueprint disabled target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2
66372025-05-08T22:08:38.270ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 21 reason = Timeout
66382025-05-08T22:08:38.270ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } dns_group = external
66392025-05-08T22:08:38.270ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 12.53\u{b5}s iteration = 12 status = {"enabled":false,"target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2"}
66402025-05-08T22:08:38.270ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.300254ms iteration = 11 status = {"addresses":["[fd00:1122:3344:101::8]:5353","[fd00:1122:3344:101::9]:5353"]}
66412025-05-08T22:08:38.270ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
66422025-05-08T22:08:38.270ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.297034ms iteration = 11 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:1::1]:5353"]}
66432025-05-08T22:08:38.270ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
66442025-05-08T22:08:38.272ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
66452025-05-08T22:08:38.272ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66462025-05-08T22:08:38.272ZDEBGnexus (BackgroundTasks): activating background_task = inventory_collection iteration = 2 reason = Timeout
66472025-05-08T22:08:38.272ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 21 reason = Timeout
66482025-05-08T22:08:38.272ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
66492025-05-08T22:08:38.272ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
66502025-05-08T22:08:38.272ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
66512025-05-08T22:08:38.272ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1494, tv_nsec: 11031367 } })
66522025-05-08T22:08:38.272ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
66532025-05-08T22:08:38.272ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1494, tv_nsec: 11151228 } })
66542025-05-08T22:08:38.272ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
66552025-05-08T22:08:38.272ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1163
66562025-05-08T22:08:38.272ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163
66572025-05-08T22:08:38.272ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
66582025-05-08T22:08:38.273ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
66592025-05-08T22:08:38.273ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-05-08 21:58:38.268898552 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
66602025-05-08T22:08:38.273ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 4.411688ms iteration = 11 status = {"expiration":"2025-05-08T21:58:38.268898552Z","pruned":[]}
66612025-05-08T22:08:38.274ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
66622025-05-08T22:08:38.274ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
66632025-05-08T22:08:38.274ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
66642025-05-08T22:08:38.277ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66652025-05-08T22:08:38.277ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66662025-05-08T22:08:38.278ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66672025-05-08T22:08:38.279ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 9.002567ms iteration = 21 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
66682025-05-08T22:08:38.279ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66692025-05-08T22:08:38.280ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
66702025-05-08T22:08:38.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
66712025-05-08T22:08:38.280ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
66722025-05-08T22:08:38.280ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
66732025-05-08T22:08:38.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
66742025-05-08T22:08:38.280ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
66752025-05-08T22:08:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
66762025-05-08T22:08:38.282ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
66772025-05-08T22:08:38.282ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66782025-05-08T22:08:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
66792025-05-08T22:08:38.283ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 61 reason = Timeout
66802025-05-08T22:08:38.283ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 21 reason = Timeout
66812025-05-08T22:08:38.283ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 21 reason = Timeout
66822025-05-08T22:08:38.283ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
66832025-05-08T22:08:38.283ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
66842025-05-08T22:08:38.283ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
66852025-05-08T22:08:38.283ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 11 reason = Timeout
66862025-05-08T22:08:38.283ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:53
66872025-05-08T22:08:38.283ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 21 reason = Timeout
66882025-05-08T22:08:38.283ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
66892025-05-08T22:08:38.283ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
66902025-05-08T22:08:38.283ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 21 reason = Timeout
66912025-05-08T22:08:38.283ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
66922025-05-08T22:08:38.283ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 4 reason = Timeout
66932025-05-08T22:08:38.283ZDEBGnexus (BackgroundTasks): activating background_task = service_firewall_rule_propagation iteration = 3 reason = Timeout
66942025-05-08T22:08:38.283ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
66952025-05-08T22:08:38.283ZDEBGnexus (service-firewall-rule-progator): starting background task for service firewall rule propagation background_task = service_firewall_rule_propagation
66962025-05-08T22:08:38.283ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
66972025-05-08T22:08:38.283ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 21 reason = Timeout
66982025-05-08T22:08:38.283ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
66992025-05-08T22:08:38.283ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
67002025-05-08T22:08:38.283ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 21 reason = Timeout
67012025-05-08T22:08:38.283ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
67022025-05-08T22:08:38.283ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
67032025-05-08T22:08:38.283ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 22 reason = Timeout
67042025-05-08T22:08:38.283ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
67052025-05-08T22:08:38.283ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 11 reason = Timeout
67062025-05-08T22:08:38.283ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
67072025-05-08T22:08:38.283ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 21 reason = Timeout
67082025-05-08T22:08:38.283ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
67092025-05-08T22:08:38.283ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
67102025-05-08T22:08:38.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
67112025-05-08T22:08:38.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
67122025-05-08T22:08:38.283ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
67132025-05-08T22:08:38.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
67142025-05-08T22:08:38.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
67152025-05-08T22:08:38.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
67162025-05-08T22:08:38.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
67172025-05-08T22:08:38.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
67182025-05-08T22:08:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
67192025-05-08T22:08:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
67202025-05-08T22:08:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
67212025-05-08T22:08:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
67222025-05-08T22:08:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
67232025-05-08T22:08:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
67242025-05-08T22:08:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
67252025-05-08T22:08:38.285ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 21 reason = Timeout
67262025-05-08T22:08:38.285ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
67272025-05-08T22:08:38.285ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 21 reason = Timeout
67282025-05-08T22:08:38.285ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
67292025-05-08T22:08:38.285ZDEBGnexus (BackgroundTasks): activating background_task = webhook_dispatcher iteration = 11 reason = Timeout
67302025-05-08T22:08:38.285ZDEBGnexus (BackgroundTasks): roles background_task = webhook_dispatcher roles = RoleSet { roles: {} }
67312025-05-08T22:08:38.285ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 11 reason = Timeout
67322025-05-08T22:08:38.285ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
67332025-05-08T22:08:38.285ZDEBGnexus (BackgroundTasks): activating background_task = tuf_artifact_replication iteration = 3 reason = Timeout
67342025-05-08T22:08:38.285ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
67352025-05-08T22:08:38.285ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 11 reason = Timeout
67362025-05-08T22:08:38.285ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
67372025-05-08T22:08:38.285ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 21 reason = Timeout
67382025-05-08T22:08:38.285ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
67392025-05-08T22:08:38.285ZDEBGnexus (BackgroundTasks): activating background_task = webhook_deliverator iteration = 11 reason = Timeout
67402025-05-08T22:08:38.285ZDEBGnexus (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} }
67412025-05-08T22:08:38.285ZDEBGnexus (BackgroundTasks): activating background_task = saga_recovery iteration = 2 reason = Timeout
67422025-05-08T22:08:38.285ZDEBGnexus (SagaRecoverer): listing candidate sagas for recovery
67432025-05-08T22:08:38.285ZDEBGnexus (SagaRecoverer): roles roles = RoleSet { roles: {} }
67442025-05-08T22:08:38.285ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 21 reason = Timeout
67452025-05-08T22:08:38.286ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
67462025-05-08T22:08:38.286ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 21 reason = Timeout
67472025-05-08T22:08:38.286ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
67482025-05-08T22:08:38.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
67492025-05-08T22:08:38.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
67502025-05-08T22:08:38.286ZDEBGnexus (SagaRecoverer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000003a8a, .. }) resource = Database result = Ok(())
67512025-05-08T22:08:38.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
67522025-05-08T22:08:38.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
67532025-05-08T22:08:38.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
67542025-05-08T22:08:38.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
67552025-05-08T22:08:38.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database result = Ok(())
67562025-05-08T22:08:38.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database result = Ok(())
67572025-05-08T22:08:38.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
67582025-05-08T22:08:38.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
67592025-05-08T22:08:38.287ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
67602025-05-08T22:08:38.287ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
67612025-05-08T22:08:38.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
67622025-05-08T22:08:38.290ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67632025-05-08T22:08:38.291ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67642025-05-08T22:08:38.293ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
67652025-05-08T22:08:38.293ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
67662025-05-08T22:08:38.293ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
67672025-05-08T22:08:38.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
67682025-05-08T22:08:38.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
67692025-05-08T22:08:38.296ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Inventory result = Ok(())
67702025-05-08T22:08:38.296ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
67712025-05-08T22:08:38.296ZDEBGnexus (BackgroundTasks): inventory_prune_one: eligible for removal background_task = inventory_collection candidates = [(9da2ae01-2248-4ec1-b67f-272a1b57dfaa, 0), (f80a208e-1e15-4655-8811-d014a87937ba, 0), (bc0b59c5-3141-4bb5-a86c-1a4b4072d11a, 0), (ce348c8a-5059-46b8-8fc3-41419986897e, 0)] collection_id = 9da2ae01-2248-4ec1-b67f-272a1b57dfaa
67722025-05-08T22:08:38.296ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
67732025-05-08T22:08:38.297ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
67742025-05-08T22:08:38.297ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
67752025-05-08T22:08:38.304ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67762025-05-08T22:08:38.308ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
67772025-05-08T22:08:38.308ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
67782025-05-08T22:08:38.308ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
67792025-05-08T22:08:38.316ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67802025-05-08T22:08:38.319ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
67812025-05-08T22:08:38.319ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
67822025-05-08T22:08:38.319ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
67832025-05-08T22:08:38.323ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 37.677603ms iteration = 11 status = {"errors":[],"found_port_ok":[]}
67842025-05-08T22:08:38.324ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
67852025-05-08T22:08:38.324ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
67862025-05-08T22:08:38.328ZDEBGnexus (BackgroundTasks): activation complete background_task = webhook_deliverator elapsed = 42.753277ms iteration = 11 status = {"by_rx":{},"error":null}
67872025-05-08T22:08:38.342ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67882025-05-08T22:08:38.343ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 57.673617ms iteration = 21 status = {"errors":[],"requests_created_ok":[]}
67892025-05-08T22:08:38.346ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
67902025-05-08T22:08:38.346ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
67912025-05-08T22:08:38.346ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
67922025-05-08T22:08:38.355ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
67932025-05-08T22:08:38.355ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
67942025-05-08T22:08:38.355ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
67952025-05-08T22:08:38.355ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
67962025-05-08T22:08:38.355ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67972025-05-08T22:08:38.355ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
67982025-05-08T22:08:38.356ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
67992025-05-08T22:08:38.356ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
68002025-05-08T22:08:38.356ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
68012025-05-08T22:08:38.356ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
68022025-05-08T22:08:38.357ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
68032025-05-08T22:08:38.361ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
68042025-05-08T22:08:38.361ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
68052025-05-08T22:08:38.361ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
68062025-05-08T22:08:38.363ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
68072025-05-08T22:08:38.363ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
68082025-05-08T22:08:38.364ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68092025-05-08T22:08:38.364ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
68102025-05-08T22:08:38.365ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
68112025-05-08T22:08:38.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
68122025-05-08T22:08:38.365ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
68132025-05-08T22:08:38.365ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
68142025-05-08T22:08:38.365ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
68152025-05-08T22:08:38.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
68162025-05-08T22:08:38.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
68172025-05-08T22:08:38.365ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
68182025-05-08T22:08:38.366ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
68192025-05-08T22:08:38.369ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
68202025-05-08T22:08:38.369ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
68212025-05-08T22:08:38.370ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
68222025-05-08T22:08:38.373ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 88.148432ms iteration = 21 status = {"errors":[],"garbage_collect_requested":[]}
68232025-05-08T22:08:38.375ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68242025-05-08T22:08:38.377ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
68252025-05-08T22:08:38.377ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68262025-05-08T22:08:38.378ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68272025-05-08T22:08:38.378ZDEBGnexus (BackgroundTasks): roles background_task = webhook_dispatcher roles = RoleSet { roles: {} }
68282025-05-08T22:08:38.378ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
68292025-05-08T22:08:38.378ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database result = Ok(())
68302025-05-08T22:08:38.378ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
68312025-05-08T22:08:38.378ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
68322025-05-08T22:08:38.379ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
68332025-05-08T22:08:38.381ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
68342025-05-08T22:08:38.381ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
68352025-05-08T22:08:38.382ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
68362025-05-08T22:08:38.382ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
68372025-05-08T22:08:38.382ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
68382025-05-08T22:08:38.382ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
68392025-05-08T22:08:38.384ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68402025-05-08T22:08:38.385ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
68412025-05-08T22:08:38.386ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
68422025-05-08T22:08:38.387ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 103.8147ms iteration = 61 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
68432025-05-08T22:08:38.387ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
68442025-05-08T22:08:38.387ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
68452025-05-08T22:08:38.387ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
68462025-05-08T22:08:38.387ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
68472025-05-08T22:08:38.387ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68482025-05-08T22:08:38.387ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
68492025-05-08T22:08:38.388ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 102.593377ms iteration = 21 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
68502025-05-08T22:08:38.390ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
68512025-05-08T22:08:38.390ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
68522025-05-08T22:08:38.390ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 104.883692ms iteration = 11 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
68532025-05-08T22:08:38.390ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
68542025-05-08T22:08:38.391ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
68552025-05-08T22:08:38.391ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
68562025-05-08T22:08:38.391ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
68572025-05-08T22:08:38.391ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
68582025-05-08T22:08:38.393ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68592025-05-08T22:08:38.394ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
68602025-05-08T22:08:38.394ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
68612025-05-08T22:08:38.394ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
68622025-05-08T22:08:38.395ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
68632025-05-08T22:08:38.396ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
68642025-05-08T22:08:38.396ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
68652025-05-08T22:08:38.396ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
68662025-05-08T22:08:38.399ZDEBGnexus (BackgroundTasks): activation complete background_task = webhook_dispatcher elapsed = 113.404173ms iteration = 11 status = {"dispatched":[],"errors":[],"glob_version":"140.0.0","globs_reprocessed":{},"no_receivers":[]}
68672025-05-08T22:08:38.399ZINFOnexus (SagaRecoverer): listed in-progress sagas count = 0 file = nexus/src/app/background/tasks/saga_recovery.rs:438
68682025-05-08T22:08:38.399ZDEBGnexus (BackgroundTasks): activation complete background_task = saga_recovery elapsed = 113.421933ms iteration = 2 status = {"last_pass":{"Success":{"nfailed":0,"nfound":0,"nrecovered":0,"nremoved":0,"nskipped":0}},"ntotal_failures":0,"ntotal_finished":0,"ntotal_recovered":0,"ntotal_sec_errors_bad_state":0,"ntotal_sec_errors_missing":0,"ntotal_started":0,"recent_failures":[],"recent_recoveries":[]}
68692025-05-08T22:08:38.401ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1494, tv_nsec: 140220798 } })
68702025-05-08T22:08:38.402ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
68712025-05-08T22:08:38.402ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68722025-05-08T22:08:38.404ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
68732025-05-08T22:08:38.404ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
68742025-05-08T22:08:38.405ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
68752025-05-08T22:08:38.407ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
68762025-05-08T22:08:38.407ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:552 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
68772025-05-08T22:08:38.407ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 123.782094ms iteration = 21 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
68782025-05-08T22:08:38.407ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
68792025-05-08T22:08:38.407ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 121.972364ms iteration = 21 status = {"errors":[],"finish_invoked_ok":[]}
68802025-05-08T22:08:38.409ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
68812025-05-08T22:08:38.410ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68822025-05-08T22:08:38.414ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
68832025-05-08T22:08:38.414ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
68842025-05-08T22:08:38.414ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
68852025-05-08T22:08:38.423ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68862025-05-08T22:08:38.425ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
68872025-05-08T22:08:38.425ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
68882025-05-08T22:08:38.425ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3753e36-4cb8-41d7-a50a-26971017df25", "content-length": "26", "date": "Thu, 08 May 2025 22:08:38 GMT"} })
68892025-05-08T22:08:38.425ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
68902025-05-08T22:08:38.426ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
68912025-05-08T22:08:38.426ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
68922025-05-08T22:08:38.426ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
68932025-05-08T22:08:38.426ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
68942025-05-08T22:08:38.428ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
68952025-05-08T22:08:38.429ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
68962025-05-08T22:08:38.429ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
68972025-05-08T22:08:38.429ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 159.552276ms iteration = 61 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
68982025-05-08T22:08:38.429ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
68992025-05-08T22:08:38.432ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69002025-05-08T22:08:38.434ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
69012025-05-08T22:08:38.434ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
69022025-05-08T22:08:38.434ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
69032025-05-08T22:08:38.439ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
69042025-05-08T22:08:38.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
69052025-05-08T22:08:38.440ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
69062025-05-08T22:08:38.440ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:137 found = 0 not_ready_to_be_deleted = 0
69072025-05-08T22:08:38.440ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 156.878268ms iteration = 11 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
69082025-05-08T22:08:38.441ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
69092025-05-08T22:08:38.441ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = d8128696-db5a-4b88-914a-69b28494e50e dns_zone_name = oxide.test found_so_far = 0 version = 2
69102025-05-08T22:08:38.441ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
69112025-05-08T22:08:38.442ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
69122025-05-08T22:08:38.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
69132025-05-08T22:08:38.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
69142025-05-08T22:08:38.445ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69152025-05-08T22:08:38.445ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 161.997403ms iteration = 21 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
69162025-05-08T22:08:38.450ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 2f2f2c32-8d58-4801-a669-9618eac0c3f7, lookup_type: ById(2f2f2c32-8d58-4801-a669-9618eac0c3f7) } result = Ok(())
69172025-05-08T22:08:38.450ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
69182025-05-08T22:08:38.450ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
69192025-05-08T22:08:38.452ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 169.380471ms iteration = 21 status = 0
69202025-05-08T22:08:38.456ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
69212025-05-08T22:08:38.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
69222025-05-08T22:08:38.460ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
69232025-05-08T22:08:38.460ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
69242025-05-08T22:08:38.460ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 177.246826ms iteration = 21 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
69252025-05-08T22:08:38.463ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
69262025-05-08T22:08:38.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
69272025-05-08T22:08:38.465ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69282025-05-08T22:08:38.465ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
69292025-05-08T22:08:38.466ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
69302025-05-08T22:08:38.469ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
69312025-05-08T22:08:38.469ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
69322025-05-08T22:08:38.469ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
69332025-05-08T22:08:38.471ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
69342025-05-08T22:08:38.471ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 7586992a-6fc5-4073-8962-728778f16aa5 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
69352025-05-08T22:08:38.471ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
69362025-05-08T22:08:38.471ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
69372025-05-08T22:08:38.474ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69382025-05-08T22:08:38.477ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
69392025-05-08T22:08:38.477ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
69402025-05-08T22:08:38.478ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "d2e97bd7-d032-4037-8577-f4e6e20cd504", "content-length": "26", "date": "Thu, 08 May 2025 22:08:38 GMT"} })
69412025-05-08T22:08:38.478ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
69422025-05-08T22:08:38.478ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
69432025-05-08T22:08:38.478ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
69442025-05-08T22:08:38.478ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
69452025-05-08T22:08:38.480ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
69462025-05-08T22:08:38.481ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 7586992a-6fc5-4073-8962-728778f16aa5 dns_zone_name = control-plane.oxide.internal found_so_far = 55 version = 1
69472025-05-08T22:08:38.481ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
69482025-05-08T22:08:38.481ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-05-08 21:59:21.182334 UTC dns_group = internal generation = 1
69492025-05-08T22:08:38.481ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 212.634585ms iteration = 11 status = {"generation":1}
69502025-05-08T22:08:38.492ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69512025-05-08T22:08:38.494ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
69522025-05-08T22:08:38.494ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
69532025-05-08T22:08:38.494ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
69542025-05-08T22:08:38.495ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
69552025-05-08T22:08:38.495ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
69562025-05-08T22:08:38.497ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
69572025-05-08T22:08:38.497ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-05-08T21:58:35.560911Z, time_modified: 2025-05-08T21:58:35.560911Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, db_silo: Silo { identity: SiloIdentity { id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-05-08T21:59:21.709488Z, time_modified: 2025-05-08T21:59:21.709488Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, db_silo: Silo { identity: SiloIdentity { id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-05-08T21:59:21.709488Z, time_modified: 2025-05-08T21:59:21.709488Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47" }] }) }
69582025-05-08T22:08:38.497ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 228.915039ms iteration = 11 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"7137ee58-6aa7-4199-a248-feadf3b8bafb","tls_certs":["37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47"]}},"default_endpoint":{"silo_id":"7137ee58-6aa7-4199-a248-feadf3b8bafb","tls_certs":["37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
69592025-05-08T22:08:38.497ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
69602025-05-08T22:08:38.498ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69612025-05-08T22:08:38.498ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
69622025-05-08T22:08:38.499ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
69632025-05-08T22:08:38.503ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
69642025-05-08T22:08:38.503ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
69652025-05-08T22:08:38.503ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
69662025-05-08T22:08:38.506ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69672025-05-08T22:08:38.506ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = d8128696-db5a-4b88-914a-69b28494e50e dns_zone_name = oxide.test found_so_far = 1 version = 2
69682025-05-08T22:08:38.506ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
69692025-05-08T22:08:38.506ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-05-08 21:59:21.759368 UTC dns_group = external generation = 2
69702025-05-08T22:08:38.506ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 237.860924ms iteration = 11 status = {"generation":2}
69712025-05-08T22:08:38.510ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 224.555682ms iteration = 21 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
69722025-05-08T22:08:38.510ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
69732025-05-08T22:08:38.510ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
69742025-05-08T22:08:38.510ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
69752025-05-08T22:08:38.525ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:362
69762025-05-08T22:08:38.525ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 253.194118ms iteration = 21 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
69772025-05-08T22:08:38.529ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
69782025-05-08T22:08:38.530ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
69792025-05-08T22:08:38.546ZINFOnexus (BackgroundTasks): removed inventory collection background_task = inventory_collection collection_id = 9da2ae01-2248-4ec1-b67f-272a1b57dfaa file = nexus/db-queries/src/db/datastore/inventory.rs:1446 ncabooses = 0 nclickhouse_keeper_membership = 0 ncollections = 0 ndatasets = 0 nerrors = 0 nnics = 0 nnvme_disk_firmware = 0 nphysical_disks = 0 nrot_pages = 0 nrots = 0 nsled_agent_zones = 0 nsled_agents = 0 nsps = 0 nzones = 0 nzpools = 0
69802025-05-08T22:08:38.546ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
69812025-05-08T22:08:38.547ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
69822025-05-08T22:08:38.579ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69832025-05-08T22:08:38.582ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
69842025-05-08T22:08:38.582ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
69852025-05-08T22:08:38.582ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
69862025-05-08T22:08:38.584ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
69872025-05-08T22:08:38.585ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "6e08d910-0be6-4991-846d-e3e3d0d569c7", "date": "Thu, 08 May 2025 22:08:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
69882025-05-08T22:08:38.585ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
69892025-05-08T22:08:38.587ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 304.068982ms iteration = 22 status = {"physical_disks_added":0}
69902025-05-08T22:08:38.587ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "89ddb31f-5c2d-4796-9446-928889eed043", "date": "Thu, 08 May 2025 22:08:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
69912025-05-08T22:08:38.587ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
69922025-05-08T22:08:38.593ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
69932025-05-08T22:08:38.593ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 324.863084ms iteration = 13 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
69942025-05-08T22:08:38.643ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e zone_address = fd00:1122:3344:101::2
69952025-05-08T22:08:38.643ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
69962025-05-08T22:08:38.644ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0550b17-e5e9-4622-94db-5b8f58946acb", "content-length": "26", "date": "Thu, 08 May 2025 22:08:38 GMT"} })
69972025-05-08T22:08:38.644ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1174 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
69982025-05-08T22:08:38.647ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
69992025-05-08T22:08:38.658ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
70002025-05-08T22:08:38.663ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 394.283467ms iteration = 21 status = {}
70012025-05-08T22:08:38.704ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
70022025-05-08T22:08:38.705ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
70032025-05-08T22:08:38.765ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
70042025-05-08T22:08:38.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
70052025-05-08T22:08:38.772ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
70062025-05-08T22:08:38.772ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
70072025-05-08T22:08:38.773ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "f82d58db-8278-43c9-8ab4-5bf957f601a4", "content-length": "372", "date": "Thu, 08 May 2025 22:08:38 GMT"} })
70082025-05-08T22:08:38.773ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
70092025-05-08T22:08:38.773ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
70102025-05-08T22:08:38.774ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70112025-05-08T22:08:38.777ZINFOnexus (BackgroundTasks): ensured all Crucible datasets present in inventory have database records background_task = blueprint_rendezvous file = nexus/reconfigurator/rendezvous/src/crucible_dataset.rs:113 num_already_exist = 9 num_inserted = 0 num_not_in_inventory = 0
70122025-05-08T22:08:38.777ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 494.433338ms iteration = 4 status = {"blueprint_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","inventory_collection_id":"ce348c8a-5059-46b8-8fc3-41419986897e","stats":{"crucible_dataset":{"num_already_exist":9,"num_inserted":0,"num_not_in_inventory":0},"debug_dataset":{"num_already_exist":9,"num_already_tombstoned":0,"num_inserted":0,"num_not_in_inventory":0,"num_tombstoned":0}}}
70132025-05-08T22:08:38.780ZDEBGnexus (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(f80a208e-1e15-4655-8811-d014a87937ba, 0), (bc0b59c5-3141-4bb5-a86c-1a4b4072d11a, 0), (ce348c8a-5059-46b8-8fc3-41419986897e, 0)]
70142025-05-08T22:08:38.781ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12225, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1494, tv_nsec: 519735857 } })
70152025-05-08T22:08:38.782ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
70162025-05-08T22:08:38.782ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
70172025-05-08T22:08:38.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
70182025-05-08T22:08:38.784ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70192025-05-08T22:08:38.785ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70202025-05-08T22:08:38.785ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
70212025-05-08T22:08:38.785ZINFOnexus (BackgroundTasks): instance reincarnation completed successfully background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:113 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0
70222025-05-08T22:08:38.785ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 502.340693ms iteration = 11 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
70232025-05-08T22:08:38.786ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
70242025-05-08T22:08:38.787ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
70252025-05-08T22:08:38.787ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
70262025-05-08T22:08:38.788ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
70272025-05-08T22:08:38.789ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
70282025-05-08T22:08:38.789ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
70292025-05-08T22:08:38.789ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
70302025-05-08T22:08:38.789ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
70312025-05-08T22:08:38.790ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
70322025-05-08T22:08:38.790ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70332025-05-08T22:08:38.790ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
70342025-05-08T22:08:38.791ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
70352025-05-08T22:08:38.791ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70362025-05-08T22:08:38.792ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70372025-05-08T22:08:38.793ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
70382025-05-08T22:08:38.793ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
70392025-05-08T22:08:38.793ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
70402025-05-08T22:08:38.795ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
70412025-05-08T22:08:38.795ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
70422025-05-08T22:08:38.796ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
70432025-05-08T22:08:38.797ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
70442025-05-08T22:08:38.797ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
70452025-05-08T22:08:38.797ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
70462025-05-08T22:08:38.800ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
70472025-05-08T22:08:38.801ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
70482025-05-08T22:08:38.802ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
70492025-05-08T22:08:38.802ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
70502025-05-08T22:08:38.804ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
70512025-05-08T22:08:38.805ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
70522025-05-08T22:08:38.806ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
70532025-05-08T22:08:38.806ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
70542025-05-08T22:08:38.809ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
70552025-05-08T22:08:38.809ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
70562025-05-08T22:08:38.809ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
70572025-05-08T22:08:38.809ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
70582025-05-08T22:08:38.810ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
70592025-05-08T22:08:38.810ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
70602025-05-08T22:08:38.811ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 543.051338ms iteration = 21 status = null
70612025-05-08T22:08:38.812ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70622025-05-08T22:08:38.815ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
70632025-05-08T22:08:38.815ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
70642025-05-08T22:08:38.815ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
70652025-05-08T22:08:38.823ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
70662025-05-08T22:08:38.823ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
70672025-05-08T22:08:38.826ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
70682025-05-08T22:08:38.826ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
70692025-05-08T22:08:38.829ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
70702025-05-08T22:08:38.829ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
70712025-05-08T22:08:38.831ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
70722025-05-08T22:08:38.832ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "b05b4813-a98f-4f27-b41d-39cdc9b6ab0d", "date": "Thu, 08 May 2025 22:08:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
70732025-05-08T22:08:38.832ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
70742025-05-08T22:08:38.832ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
70752025-05-08T22:08:38.833ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
70762025-05-08T22:08:38.834ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "18332f92-13a0-46f4-9487-f17173403202", "date": "Thu, 08 May 2025 22:08:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
70772025-05-08T22:08:38.834ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
70782025-05-08T22:08:38.834ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
70792025-05-08T22:08:38.834ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
70802025-05-08T22:08:38.835ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
70812025-05-08T22:08:38.835ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
70822025-05-08T22:08:38.836ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "386086d9-b366-4ec9-b3f6-4dd4018c628a", "date": "Thu, 08 May 2025 22:08:38 GMT"} })
70832025-05-08T22:08:38.836ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
70842025-05-08T22:08:38.836ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "b0fca57b-d23c-4938-b70a-2ff79f1d44ee", "date": "Thu, 08 May 2025 22:08:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
70852025-05-08T22:08:38.836ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
70862025-05-08T22:08:38.836ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
70872025-05-08T22:08:38.836ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
70882025-05-08T22:08:38.837ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
70892025-05-08T22:08:38.838ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
70902025-05-08T22:08:38.838ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 569.698753ms iteration = 13 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
70912025-05-08T22:08:38.838ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70922025-05-08T22:08:38.839ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
70932025-05-08T22:08:38.839ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
70942025-05-08T22:08:38.842ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
70952025-05-08T22:08:38.842ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
70962025-05-08T22:08:38.843ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
70972025-05-08T22:08:38.843ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
70982025-05-08T22:08:38.843ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
70992025-05-08T22:08:38.844ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
71002025-05-08T22:08:38.844ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
71012025-05-08T22:08:38.845ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "95bf5e95-d9fb-4d7e-a7ab-f5101fabf7e1", "date": "Thu, 08 May 2025 22:08:38 GMT"} })
71022025-05-08T22:08:38.845ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
71032025-05-08T22:08:38.845ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
71042025-05-08T22:08:38.847ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 563.79622ms iteration = 21 status = {}
71052025-05-08T22:08:38.847ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
71062025-05-08T22:08:38.847ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
71072025-05-08T22:08:38.849ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71082025-05-08T22:08:38.854ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
71092025-05-08T22:08:38.854ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
71102025-05-08T22:08:38.854ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
71112025-05-08T22:08:38.858ZDEBGnexus (BackgroundTasks): begin collection background_task = inventory_collection
71122025-05-08T22:08:38.858ZDEBGnexus (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[fd00:1122:3344:101::2]:12225
71132025-05-08T22:08:38.858ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/ignition
71142025-05-08T22:08:38.859ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "af73fada-b5a5-46d0-9f34-a47df4b15399", "content-length": "476", "date": "Thu, 08 May 2025 22:08:38 GMT"} })
71152025-05-08T22:08:38.859ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0
71162025-05-08T22:08:38.859ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "49096501-2939-4b1f-b795-47f9e46c3e88", "content-length": "734", "date": "Thu, 08 May 2025 22:08:38 GMT"} })
71172025-05-08T22:08:38.859ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=0
71182025-05-08T22:08:38.860ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4acfd7a-587c-4705-9e77-dacad0e6a6db", "content-length": "111", "date": "Thu, 08 May 2025 22:08:38 GMT"} })
71192025-05-08T22:08:38.860ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=1
71202025-05-08T22:08:38.860ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dcb9f478-37b4-40c3-a25f-c77430f30a44", "content-length": "111", "date": "Thu, 08 May 2025 22:08:38 GMT"} })
71212025-05-08T22:08:38.860ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=0
71222025-05-08T22:08:38.860ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "347a77d7-6baa-4ccf-a272-bca71595e989", "content-length": "170", "date": "Thu, 08 May 2025 22:08:38 GMT"} })
71232025-05-08T22:08:38.860ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=1
71242025-05-08T22:08:38.860ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0dd61d03-6e20-4fbc-9d68-2173eb46f898", "content-length": "170", "date": "Thu, 08 May 2025 22:08:38 GMT"} })
71252025-05-08T22:08:38.860ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=0
71262025-05-08T22:08:38.861ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2226d9f8-195f-4c2f-9f59-b112ba311632", "content-length": "179", "date": "Thu, 08 May 2025 22:08:38 GMT"} })
71272025-05-08T22:08:38.861ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=1
71282025-05-08T22:08:38.861ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b28bc8d9-e897-48cd-a4e8-a9c856fc675b", "content-length": "179", "date": "Thu, 08 May 2025 22:08:38 GMT"} })
71292025-05-08T22:08:38.861ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cmpa
71302025-05-08T22:08:38.861ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7de82aea-c026-4573-b673-13f0ece3ec0a", "content-length": "702", "date": "Thu, 08 May 2025 22:08:38 GMT"} })
71312025-05-08T22:08:38.861ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa
71322025-05-08T22:08:38.861ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6efde5ce-1284-4f03-999d-a445c4262718", "content-length": "727", "date": "Thu, 08 May 2025 22:08:38 GMT"} })
71332025-05-08T22:08:38.861ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa
71342025-05-08T22:08:38.861ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "206c82bb-e082-4833-b37d-0e5c25428f7f", "content-length": "729", "date": "Thu, 08 May 2025 22:08:38 GMT"} })
71352025-05-08T22:08:38.861ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa
71362025-05-08T22:08:38.862ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "643fd924-d836-4825-8032-907d03fe8337", "content-length": "728", "date": "Thu, 08 May 2025 22:08:38 GMT"} })
71372025-05-08T22:08:38.862ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0
71382025-05-08T22:08:38.862ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5344a34a-1ea8-4ffc-af76-10107fe6e432", "content-length": "722", "date": "Thu, 08 May 2025 22:08:38 GMT"} })
71392025-05-08T22:08:38.862ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=0
71402025-05-08T22:08:38.862ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9fd7a3f5-17f3-4cee-995c-012910145089", "content-length": "109", "date": "Thu, 08 May 2025 22:08:38 GMT"} })
71412025-05-08T22:08:38.862ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=1
71422025-05-08T22:08:38.862ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "02d45cdc-94dd-4444-b1b8-ec079c03a541", "content-length": "109", "date": "Thu, 08 May 2025 22:08:38 GMT"} })
71432025-05-08T22:08:38.862ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=0
71442025-05-08T22:08:38.863ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fec883fc-40ab-45d0-ab23-5549e617759c", "content-length": "169", "date": "Thu, 08 May 2025 22:08:38 GMT"} })
71452025-05-08T22:08:38.863ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=1
71462025-05-08T22:08:38.863ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6692cb90-a787-4cc9-b348-145460073705", "content-length": "169", "date": "Thu, 08 May 2025 22:08:38 GMT"} })
71472025-05-08T22:08:38.863ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=0
71482025-05-08T22:08:38.863ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
71492025-05-08T22:08:38.863ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4bd5915c-a0ee-4de7-9778-c79afa42eb34", "content-length": "178", "date": "Thu, 08 May 2025 22:08:38 GMT"} })
71502025-05-08T22:08:38.863ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=1
71512025-05-08T22:08:38.864ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
71522025-05-08T22:08:38.864ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a32242f6-6152-45ec-bc4c-c7f07cec3d57", "content-length": "178", "date": "Thu, 08 May 2025 22:08:38 GMT"} })
71532025-05-08T22:08:38.864ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cmpa
71542025-05-08T22:08:38.864ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0059eab1-fb99-49f4-807d-8cddd1fefd20", "content-length": "702", "date": "Thu, 08 May 2025 22:08:38 GMT"} })
71552025-05-08T22:08:38.864ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa
71562025-05-08T22:08:38.864ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d0814aba-a16a-4855-9e99-a6276011c939", "content-length": "727", "date": "Thu, 08 May 2025 22:08:38 GMT"} })
71572025-05-08T22:08:38.864ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa
71582025-05-08T22:08:38.865ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b049f2f4-f403-4815-b29f-2deb919072d3", "content-length": "729", "date": "Thu, 08 May 2025 22:08:38 GMT"} })
71592025-05-08T22:08:38.865ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa
71602025-05-08T22:08:38.866ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "06072721-33c3-46f5-890f-4772fb246300", "content-length": "728", "date": "Thu, 08 May 2025 22:08:38 GMT"} })
71612025-05-08T22:08:38.866ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
71622025-05-08T22:08:38.866ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
71632025-05-08T22:08:38.866ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
71642025-05-08T22:08:38.866ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
71652025-05-08T22:08:38.869ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
71662025-05-08T22:08:38.869ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71672025-05-08T22:08:38.869ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
71682025-05-08T22:08:38.872ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
71692025-05-08T22:08:38.872ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
71702025-05-08T22:08:38.872ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
71712025-05-08T22:08:38.872ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
71722025-05-08T22:08:38.872ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
71732025-05-08T22:08:38.874ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71742025-05-08T22:08:38.875ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
71752025-05-08T22:08:38.875ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
71762025-05-08T22:08:38.876ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
71772025-05-08T22:08:38.876ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
71782025-05-08T22:08:38.877ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
71792025-05-08T22:08:38.912ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
71802025-05-08T22:08:38.912ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = tuf_artifact_replication body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/artifacts-config
71812025-05-08T22:08:38.914ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
71822025-05-08T22:08:38.914ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
71832025-05-08T22:08:38.916ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
71842025-05-08T22:08:38.916ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
71852025-05-08T22:08:38.918ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71862025-05-08T22:08:38.927ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
71872025-05-08T22:08:38.927ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
71882025-05-08T22:08:38.927ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
71892025-05-08T22:08:38.928ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
71902025-05-08T22:08:38.928ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
71912025-05-08T22:08:38.929ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
71922025-05-08T22:08:38.929ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
71932025-05-08T22:08:38.931ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: cd0b930c-534b-46ad-ac3b-2eaa94b0bf2c, rack_id: 64f3d6f4-89b8-48aa-a622-b684e261667e, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(65f83a36-18fb-49b7-9914-87c8517d200e) } rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
71942025-05-08T22:08:38.931ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
71952025-05-08T22:08:38.931ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
71962025-05-08T22:08:38.931ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
71972025-05-08T22:08:38.932ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
71982025-05-08T22:08:38.933ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71992025-05-08T22:08:38.940ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
72002025-05-08T22:08:38.940ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
72012025-05-08T22:08:38.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
72022025-05-08T22:08:38.944ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1833 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
72032025-05-08T22:08:38.944ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1860 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 65f83a36-18fb-49b7-9914-87c8517d200e
72042025-05-08T22:08:38.944ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
72052025-05-08T22:08:38.944ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
72062025-05-08T22:08:38.944ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
72072025-05-08T22:08:38.944ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
72082025-05-08T22:08:38.944ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
72092025-05-08T22:08:38.944ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2007 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
72102025-05-08T22:08:38.944ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
72112025-05-08T22:08:38.944ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2043 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
72122025-05-08T22:08:38.944ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
72132025-05-08T22:08:38.944ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
72142025-05-08T22:08:38.945ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
72152025-05-08T22:08:38.947ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
72162025-05-08T22:08:38.947ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
72172025-05-08T22:08:38.948ZDEBGnexus (service-firewall-rule-progator): resolved names for firewall rules background_task = service_firewall_rule_propagation instance_interfaces = {} subnet_interfaces = {Name("nexus"): [NetworkInterface { id: 559b827c-9574-475a-93d6-1045f64398e6, kind: Service { id: 85a39cf9-9419-458f-af2e-c8b40d8f5f79 }, name: Name("nexus-85a39cf9-9419-458f-af2e-c8b40d8f5f79"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 232, 9])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 715a8bac-a423-4460-aac2-b8c76af6bb34, kind: Service { id: ddd82634-2844-4f97-9746-07099044c19d }, name: Name("nexus-ddd82634-2844-4f97-9746-07099044c19d"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 227, 84])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 76144c16-58bd-4abd-b758-565497d9f6b8, kind: Service { id: 53ef6679-65e8-4486-9401-d6fa4ff2e8e4 }, name: Name("nexus-53ef6679-65e8-4486-9401-d6fa4ff2e8e4"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 174, 62])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], Name("external-dns"): [NetworkInterface { id: 517232ae-3fd2-40fd-801d-796bf5ecd14e, kind: Service { id: 0ccabf41-81cd-45e7-bb26-335edaa2ee47 }, name: Name("external-dns-0ccabf41-81cd-45e7-bb26-335edaa2ee47"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 224, 221])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: b845b0e9-661e-4fa1-9292-0c7dce84b88d, kind: Service { id: 38a08dd6-6a75-4558-b4f8-df853f4d03d9 }, name: Name("external-dns-38a08dd6-6a75-4558-b4f8-df853f4d03d9"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 179, 43])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }]} subnet_networks = {Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })], Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })]} vpc_interfaces = {}
72182025-05-08T22:08:38.948ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
72192025-05-08T22:08:38.949ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
72202025-05-08T22:08:38.950ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72212025-05-08T22:08:38.952ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Fleet result = Ok(())
72222025-05-08T22:08:38.952ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
72232025-05-08T22:08:38.952ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
72242025-05-08T22:08:38.954ZDEBGnexus (service-firewall-rule-progator): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified. background_task = service_firewall_rule_propagation
72252025-05-08T22:08:38.954ZDEBGnexus (service-firewall-rule-progator): resolved firewall rules for sled agents background_task = service_firewall_rule_propagation sled_agent_rules = [ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("53")]), filter_protocols: Some([Udp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 517232ae-3fd2-40fd-801d-796bf5ecd14e, kind: Service { id: 0ccabf41-81cd-45e7-bb26-335edaa2ee47 }, name: Name("external-dns-0ccabf41-81cd-45e7-bb26-335edaa2ee47"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 224, 221])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: b845b0e9-661e-4fa1-9292-0c7dce84b88d, kind: Service { id: 38a08dd6-6a75-4558-b4f8-df853f4d03d9 }, name: Name("external-dns-38a08dd6-6a75-4558-b4f8-df853f4d03d9"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 179, 43])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }, ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("80"), L4PortRange("443")]), filter_protocols: Some([Tcp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 559b827c-9574-475a-93d6-1045f64398e6, kind: Service { id: 85a39cf9-9419-458f-af2e-c8b40d8f5f79 }, name: Name("nexus-85a39cf9-9419-458f-af2e-c8b40d8f5f79"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 232, 9])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 715a8bac-a423-4460-aac2-b8c76af6bb34, kind: Service { id: ddd82634-2844-4f97-9746-07099044c19d }, name: Name("nexus-ddd82634-2844-4f97-9746-07099044c19d"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 227, 84])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 76144c16-58bd-4abd-b758-565497d9f6b8, kind: Service { id: 53ef6679-65e8-4486-9401-d6fa4ff2e8e4 }, name: Name("nexus-53ef6679-65e8-4486-9401-d6fa4ff2e8e4"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 174, 62])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }]
72262025-05-08T22:08:38.954ZDEBGnexus (service-firewall-rule-progator): resolved 2 rules for sleds background_task = service_firewall_rule_propagation
72272025-05-08T22:08:38.994ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "d01779d2-3516-49a0-901a-bc2436734362", "content-length": "691", "date": "Thu, 08 May 2025 22:08:38 GMT"} })
72282025-05-08T22:08:38.994ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
72292025-05-08T22:08:38.994ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
72302025-05-08T22:08:38.994ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
72312025-05-08T22:08:38.995ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 712.396969ms iteration = 21 status = {}
72322025-05-08T22:08:39.002ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = tuf_artifact_replication result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/artifacts-config", status: 204, headers: {"x-request-id": "898c1d5a-5132-4fd5-96b9-2727045125e3", "date": "Thu, 08 May 2025 22:08:38 GMT"} })
72332025-05-08T22:08:39.002ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = tuf_artifact_replication body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/artifacts
72342025-05-08T22:08:39.003ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = tuf_artifact_replication result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/artifacts", status: 200, headers: {"content-type": "application/json", "x-request-id": "7424d100-60b1-438b-91f3-ead4fcbc432f", "content-length": "26", "date": "Thu, 08 May 2025 22:08:38 GMT"} })
72352025-05-08T22:08:39.003ZINFOnexus (BackgroundTasks): Successfully got artifact list background_task = tuf_artifact_replication file = nexus/src/app/background/tasks/tuf_artifact_replication.rs:673 sled = http://[fd00:1122:3344:101::1]:12345
72362025-05-08T22:08:39.004ZDEBGnexus (BackgroundTasks): activation complete background_task = tuf_artifact_replication elapsed = 719.153722ms iteration = 3 status = {"generation":1,"last_run_counters":{"copy_err":0,"copy_ok":0,"list_err":0,"list_ok":1,"put_config_err":0,"put_config_ok":1,"put_err":0,"put_ok":0},"lifetime_counters":{"copy_err":0,"copy_ok":0,"list_err":0,"list_ok":2,"put_config_err":0,"put_config_ok":2,"put_err":0,"put_ok":0},"local_repos":0,"request_debug_ringbuf":[{"generation":1,"operation":"put_config","target_sled":"2f2f2c32-8d58-4801-a669-9618eac0c3f7","time":"2025-05-08T22:03:39.017442287Z"},{"operation":"list","target_sled":"2f2f2c32-8d58-4801-a669-9618eac0c3f7","time":"2025-05-08T22:03:39.022391931Z"},{"generation":1,"operation":"put_config","target_sled":"2f2f2c32-8d58-4801-a669-9618eac0c3f7","time":"2025-05-08T22:08:39.002705581Z"},{"operation":"list","target_sled":"2f2f2c32-8d58-4801-a669-9618eac0c3f7","time":"2025-05-08T22:08:39.003341248Z"}]}
72372025-05-08T22:08:39.019ZDEBGnexus (service-firewall-rule-progator): resolved sleds for vpc oxide-services background_task = service_firewall_rule_propagation vpc_to_sled = [Sled { identity: SledIdentity { id: 2f2f2c32-8d58-4801-a669-9618eac0c3f7, time_created: 2025-05-08T21:59:12.125305Z, time_modified: 2025-05-08T21:59:12.125305Z }, time_deleted: None, rcgen: Generation(Generation(19)), rack_id: 64f3d6f4-89b8-48aa-a622-b684e261667e, is_scrimlet: true, serial_number: "sock", part_number: "i86pc", revision: SqlU32(0), usable_hardware_threads: SqlU32(48), usable_physical_ram: ByteCount(ByteCount(102952026112)), reservoir_size: ByteCount(ByteCount(56094621696)), ip: fd00:1122:3344:101::1, port: SqlU16(12345), last_used_address: fd00:1122:3344:101::ffff, policy: InService, state: Active, sled_agent_gen: Generation(Generation(1)), repo_depot_port: SqlU16(12348) }]
72382025-05-08T22:08:39.019ZDEBGnexus (service-firewall-rule-progator): sending firewall rules to sled agents background_task = service_firewall_rule_propagation
72392025-05-08T22:08:39.019ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
72402025-05-08T22:08:39.019ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
72412025-05-08T22:08:39.022ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
72422025-05-08T22:08:39.023ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
72432025-05-08T22:08:39.035ZDEBGnexus (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[fd00:1122:3344:101::1]:12345
72442025-05-08T22:08:39.035ZDEBGnexus (BackgroundTasks): client request SledAgent = http://[fd00:1122:3344:101::1]:12345 background_task = inventory_collection body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/inventory
72452025-05-08T22:08:39.037ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72462025-05-08T22:08:39.041ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Sled { parent: Fleet, key: 2f2f2c32-8d58-4801-a669-9618eac0c3f7, lookup_type: ById(2f2f2c32-8d58-4801-a669-9618eac0c3f7) } result = Ok(())
72472025-05-08T22:08:39.073ZDEBGnexus (service-firewall-rule-progator): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = service_firewall_rule_propagation body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules
72482025-05-08T22:08:39.077ZDEBGnexus (service-firewall-rule-progator): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = service_firewall_rule_propagation result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules", status: 204, headers: {"x-request-id": "49fded1b-4f81-4785-aa36-dee74634ba21", "date": "Thu, 08 May 2025 22:08:38 GMT"} })
72492025-05-08T22:08:39.078ZINFOnexus (service-firewall-rule-progator): updated firewall rules on 1 sleds background_task = service_firewall_rule_propagation file = nexus/networking/src/firewall_rules.rs:495
72502025-05-08T22:08:39.078ZDEBGnexus (service-firewall-rule-progator): successfully propagated service firewall rules background_task = service_firewall_rule_propagation elapsed = 795.307237ms
72512025-05-08T22:08:39.078ZDEBGnexus (BackgroundTasks): activation complete background_task = service_firewall_rule_propagation elapsed = 795.318517ms iteration = 3 status = {}
72522025-05-08T22:08:39.148ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "bf0e430c-6891-4231-9fe8-82f0cee46b23", "date": "Thu, 08 May 2025 22:08:38 GMT"} })
72532025-05-08T22:08:39.148ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch = Switch0
72542025-05-08T22:08:39.148ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
72552025-05-08T22:08:39.148ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e1414bf-9f78-41b9-b21c-70ee11cdc457", "content-length": "682", "date": "Thu, 08 May 2025 22:08:38 GMT"} })
72562025-05-08T22:08:39.148ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
72572025-05-08T22:08:39.149ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
72582025-05-08T22:08:39.153ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
72592025-05-08T22:08:39.153ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
72602025-05-08T22:08:39.156ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
72612025-05-08T22:08:39.156ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
72622025-05-08T22:08:39.158ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
72632025-05-08T22:08:39.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
72642025-05-08T22:08:39.165ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 882.223107ms iteration = 21 status = {}
72652025-05-08T22:08:39.493ZDEBGnexus (BackgroundTasks): client response SledAgent = http://[fd00:1122:3344:101::1]:12345 background_task = inventory_collection result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "b2726879-cc1a-456b-8c88-fc78349e574f", "content-length": "31972", "date": "Thu, 08 May 2025 22:08:39 GMT"} })
72662025-05-08T22:08:39.523ZDEBGnexus (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
72672025-05-08T22:08:39.523ZDEBGnexus (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
72682025-05-08T22:08:39.523ZDEBGnexus (BackgroundTasks): finished collection background_task = inventory_collection
72692025-05-08T22:08:39.527ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
72702025-05-08T22:08:39.527ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
72712025-05-08T22:08:39.529ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72722025-05-08T22:08:39.531ZDEBGnexus (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
72732025-05-08T22:08:39.531ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
72742025-05-08T22:08:39.531ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
72752025-05-08T22:08:39.686ZINFOnexus (BackgroundTasks): inserted inventory collection background_task = inventory_collection collection_id = 50753503-b10e-4df9-a530-a8658be2e71a file = nexus/db-queries/src/db/datastore/inventory.rs:1009
72762025-05-08T22:08:39.686ZDEBGnexus (BackgroundTasks): inventory collection complete background_task = inventory_collection collection_id = 50753503-b10e-4df9-a530-a8658be2e71a time_started = 2025-05-08 22:08:38.858834 UTC
72772025-05-08T22:08:39.686ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 23 reason = Dependency
72782025-05-08T22:08:39.686ZDEBGnexus (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.414645251s iteration = 2 status = {"collection_id":"50753503-b10e-4df9-a530-a8658be2e71a","time_done":"2025-05-08 22:08:39.525405 UTC","time_started":"2025-05-08 22:08:38.858834 UTC"}
72792025-05-08T22:08:39.686ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 5 reason = Dependency
72802025-05-08T22:08:39.686ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
72812025-05-08T22:08:39.686ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
72822025-05-08T22:08:39.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
72832025-05-08T22:08:39.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
72842025-05-08T22:08:39.688ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72852025-05-08T22:08:39.689ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72862025-05-08T22:08:39.692ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
72872025-05-08T22:08:39.692ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
72882025-05-08T22:08:39.692ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
72892025-05-08T22:08:39.692ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Inventory result = Ok(())
72902025-05-08T22:08:39.692ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
72912025-05-08T22:08:39.692ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
72922025-05-08T22:08:39.694ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
72932025-05-08T22:08:39.694ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
72942025-05-08T22:08:39.696ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72952025-05-08T22:08:39.698ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
72962025-05-08T22:08:39.698ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
72972025-05-08T22:08:39.698ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
72982025-05-08T22:08:39.727ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 41.18027ms iteration = 23 status = {"physical_disks_added":0}
72992025-05-08T22:08:39.735ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
73002025-05-08T22:08:39.735ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
73012025-05-08T22:08:39.765ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
73022025-05-08T22:08:39.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
73032025-05-08T22:08:39.767ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73042025-05-08T22:08:39.769ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
73052025-05-08T22:08:39.769ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
73062025-05-08T22:08:39.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
73072025-05-08T22:08:39.770ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73082025-05-08T22:08:39.772ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
73092025-05-08T22:08:39.772ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
73102025-05-08T22:08:39.772ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
73112025-05-08T22:08:39.775ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
73122025-05-08T22:08:39.775ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
73132025-05-08T22:08:39.777ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73142025-05-08T22:08:39.779ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
73152025-05-08T22:08:39.779ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
73162025-05-08T22:08:39.779ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
73172025-05-08T22:08:39.780ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73182025-05-08T22:08:39.782ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
73192025-05-08T22:08:39.782ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
73202025-05-08T22:08:39.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
73212025-05-08T22:08:39.784ZINFOnexus (BackgroundTasks): ensured all Crucible datasets present in inventory have database records background_task = blueprint_rendezvous file = nexus/reconfigurator/rendezvous/src/crucible_dataset.rs:113 num_already_exist = 9 num_inserted = 0 num_not_in_inventory = 0
73222025-05-08T22:08:39.784ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 98.12669ms iteration = 5 status = {"blueprint_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","inventory_collection_id":"b163257d-77a0-4b31-9b71-80dbd869fa80","stats":{"crucible_dataset":{"num_already_exist":9,"num_inserted":0,"num_not_in_inventory":0},"debug_dataset":{"num_already_exist":9,"num_already_tombstoned":0,"num_inserted":0,"num_not_in_inventory":0,"num_tombstoned":0}}}
73232025-05-08T22:08:41.938ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 269 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 6b0255a6-f59f-46a9-bb8e-3f88998ac0fa response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
73242025-05-08T22:08:48.270ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 62 reason = Timeout
73252025-05-08T22:08:48.270ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
73262025-05-08T22:08:48.270ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
73272025-05-08T22:08:48.272ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73282025-05-08T22:08:48.274ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
73292025-05-08T22:08:48.274ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
73302025-05-08T22:08:48.275ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
73312025-05-08T22:08:48.276ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
73322025-05-08T22:08:48.277ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
73332025-05-08T22:08:48.278ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73342025-05-08T22:08:48.281ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
73352025-05-08T22:08:48.281ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
73362025-05-08T22:08:48.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
73372025-05-08T22:08:48.283ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 62 reason = Timeout
73382025-05-08T22:08:48.283ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
73392025-05-08T22:08:48.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
73402025-05-08T22:08:48.285ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
73412025-05-08T22:08:48.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
73422025-05-08T22:08:48.287ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.375426ms iteration = 62 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
73432025-05-08T22:08:48.289ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
73442025-05-08T22:08:48.289ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 19.250916ms iteration = 62 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
73452025-05-08T22:08:51.931ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::d]:43041
73462025-05-08T22:08:51.931ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:43041 req_id = 3572a5cf-54cd-477e-96e4-25fb8d5fe7b0 roles = RoleSet { roles: {} } uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?sort_by=id_ascending
73472025-05-08T22:08:51.931ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:43041 req_id = 3572a5cf-54cd-477e-96e4-25fb8d5fe7b0 resource = Database result = Ok(()) uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?sort_by=id_ascending
73482025-05-08T22:08:51.934ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2902 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:43041 req_id = 3572a5cf-54cd-477e-96e4-25fb8d5fe7b0 response_code = 200 uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?sort_by=id_ascending
73492025-05-08T22:08:51.934ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:43041 req_id = f96494be-5f83-45c6-a450-52df4962b73a roles = RoleSet { roles: {} } uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZGQ4MjYzNC0yODQ0LTRmOTctOTc0Ni0wNzA5OTA0NGMxOWQifX0%3D
73502025-05-08T22:08:51.934ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:43041 req_id = f96494be-5f83-45c6-a450-52df4962b73a resource = Database result = Ok(()) uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZGQ4MjYzNC0yODQ0LTRmOTctOTc0Ni0wNzA5OTA0NGMxOWQifX0%3D
73512025-05-08T22:08:51.936ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2439 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:43041 req_id = f96494be-5f83-45c6-a450-52df4962b73a response_code = 200 uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZGQ4MjYzNC0yODQ0LTRmOTctOTc0Ni0wNzA5OTA0NGMxOWQifX0%3D
73522025-05-08T22:08:51.938ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 243 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 1aba56fb-baca-4f68-8052-75177281d50c response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
73532025-05-08T22:08:58.270ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 63 reason = Timeout
73542025-05-08T22:08:58.270ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
73552025-05-08T22:08:58.271ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
73562025-05-08T22:08:58.272ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73572025-05-08T22:08:58.275ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
73582025-05-08T22:08:58.275ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
73592025-05-08T22:08:58.275ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
73602025-05-08T22:08:58.277ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
73612025-05-08T22:08:58.277ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
73622025-05-08T22:08:58.279ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73632025-05-08T22:08:58.282ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
73642025-05-08T22:08:58.282ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
73652025-05-08T22:08:58.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
73662025-05-08T22:08:58.284ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 63 reason = Timeout
73672025-05-08T22:08:58.284ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
73682025-05-08T22:08:58.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
73692025-05-08T22:08:58.286ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
73702025-05-08T22:08:58.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
73712025-05-08T22:08:58.287ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.405386ms iteration = 63 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
73722025-05-08T22:08:58.291ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
73732025-05-08T22:08:58.291ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.386638ms iteration = 63 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
73742025-05-08T22:09:01.938ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 275 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 1ad16231-ae2a-43f8-9dbd-5476e4a3c26f response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
73752025-05-08T22:09:06.931ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:43041 req_id = 94293389-b89a-4d11-af3d-a55e1d3e6ed5 roles = RoleSet { roles: {} } uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?sort_by=id_ascending
73762025-05-08T22:09:06.931ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:43041 req_id = 94293389-b89a-4d11-af3d-a55e1d3e6ed5 resource = Database result = Ok(()) uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?sort_by=id_ascending
73772025-05-08T22:09:06.934ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3399 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:43041 req_id = 94293389-b89a-4d11-af3d-a55e1d3e6ed5 response_code = 200 uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?sort_by=id_ascending
73782025-05-08T22:09:06.935ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:43041 req_id = 1b74d3a3-d45e-48e3-ba1b-09be7608c02f roles = RoleSet { roles: {} } uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZGQ4MjYzNC0yODQ0LTRmOTctOTc0Ni0wNzA5OTA0NGMxOWQifX0%3D
73792025-05-08T22:09:06.935ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:43041 req_id = 1b74d3a3-d45e-48e3-ba1b-09be7608c02f resource = Database result = Ok(()) uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZGQ4MjYzNC0yODQ0LTRmOTctOTc0Ni0wNzA5OTA0NGMxOWQifX0%3D
73802025-05-08T22:09:06.937ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2356 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:43041 req_id = 1b74d3a3-d45e-48e3-ba1b-09be7608c02f response_code = 200 uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZGQ4MjYzNC0yODQ0LTRmOTctOTc0Ni0wNzA5OTA0NGMxOWQifX0%3D
73812025-05-08T22:09:08.268ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 22 reason = Timeout
73822025-05-08T22:09:08.268ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 22 reason = Timeout
73832025-05-08T22:09:08.268ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
73842025-05-08T22:09:08.268ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
73852025-05-08T22:09:08.268ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
73862025-05-08T22:09:08.269ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
73872025-05-08T22:09:08.270ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 64 reason = Timeout
73882025-05-08T22:09:08.270ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
73892025-05-08T22:09:08.270ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 22 reason = Timeout
73902025-05-08T22:09:08.270ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
73912025-05-08T22:09:08.270ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1524, tv_nsec: 9073976 } })
73922025-05-08T22:09:08.270ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163
73932025-05-08T22:09:08.271ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1524, tv_nsec: 9764833 } })
73942025-05-08T22:09:08.271ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1163
73952025-05-08T22:09:08.272ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 22 reason = Timeout
73962025-05-08T22:09:08.272ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73972025-05-08T22:09:08.272ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
73982025-05-08T22:09:08.272ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
73992025-05-08T22:09:08.273ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.66578ms iteration = 22 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
74002025-05-08T22:09:08.274ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74012025-05-08T22:09:08.275ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
74022025-05-08T22:09:08.275ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
74032025-05-08T22:09:08.275ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
74042025-05-08T22:09:08.276ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
74052025-05-08T22:09:08.276ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
74062025-05-08T22:09:08.277ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
74072025-05-08T22:09:08.277ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
74082025-05-08T22:09:08.277ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
74092025-05-08T22:09:08.278ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74102025-05-08T22:09:08.282ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
74112025-05-08T22:09:08.282ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
74122025-05-08T22:09:08.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
74132025-05-08T22:09:08.283ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 64 reason = Timeout
74142025-05-08T22:09:08.283ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
74152025-05-08T22:09:08.283ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 22 reason = Timeout
74162025-05-08T22:09:08.283ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
74172025-05-08T22:09:08.283ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
74182025-05-08T22:09:08.283ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 24 reason = Timeout
74192025-05-08T22:09:08.283ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 22 reason = Timeout
74202025-05-08T22:09:08.283ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
74212025-05-08T22:09:08.283ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
74222025-05-08T22:09:08.283ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
74232025-05-08T22:09:08.283ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 22 reason = Timeout
74242025-05-08T22:09:08.283ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 22 reason = Timeout
74252025-05-08T22:09:08.283ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
74262025-05-08T22:09:08.283ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
74272025-05-08T22:09:08.283ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 22 reason = Timeout
74282025-05-08T22:09:08.283ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 22 reason = Timeout
74292025-05-08T22:09:08.283ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
74302025-05-08T22:09:08.283ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
74312025-05-08T22:09:08.283ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 22 reason = Timeout
74322025-05-08T22:09:08.283ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
74332025-05-08T22:09:08.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
74342025-05-08T22:09:08.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
74352025-05-08T22:09:08.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
74362025-05-08T22:09:08.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
74372025-05-08T22:09:08.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
74382025-05-08T22:09:08.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
74392025-05-08T22:09:08.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
74402025-05-08T22:09:08.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
74412025-05-08T22:09:08.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
74422025-05-08T22:09:08.285ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 22 reason = Timeout
74432025-05-08T22:09:08.285ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 22 reason = Timeout
74442025-05-08T22:09:08.285ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
74452025-05-08T22:09:08.285ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
74462025-05-08T22:09:08.285ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 22 reason = Timeout
74472025-05-08T22:09:08.285ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
74482025-05-08T22:09:08.285ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 22 reason = Timeout
74492025-05-08T22:09:08.285ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
74502025-05-08T22:09:08.285ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 22 reason = Timeout
74512025-05-08T22:09:08.285ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
74522025-05-08T22:09:08.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
74532025-05-08T22:09:08.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
74542025-05-08T22:09:08.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
74552025-05-08T22:09:08.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
74562025-05-08T22:09:08.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
74572025-05-08T22:09:08.286ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74582025-05-08T22:09:08.287ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
74592025-05-08T22:09:08.287ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
74602025-05-08T22:09:08.287ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
74612025-05-08T22:09:08.287ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
74622025-05-08T22:09:08.287ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74632025-05-08T22:09:08.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
74642025-05-08T22:09:08.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
74652025-05-08T22:09:08.289ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
74662025-05-08T22:09:08.289ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
74672025-05-08T22:09:08.289ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
74682025-05-08T22:09:08.290ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
74692025-05-08T22:09:08.290ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
74702025-05-08T22:09:08.290ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74712025-05-08T22:09:08.290ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
74722025-05-08T22:09:08.292ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
74732025-05-08T22:09:08.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
74742025-05-08T22:09:08.292ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74752025-05-08T22:09:08.292ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
74762025-05-08T22:09:08.292ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
74772025-05-08T22:09:08.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
74782025-05-08T22:09:08.293ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74792025-05-08T22:09:08.295ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
74802025-05-08T22:09:08.295ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
74812025-05-08T22:09:08.295ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
74822025-05-08T22:09:08.296ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 10.158349ms iteration = 22 status = {"errors":[],"finish_invoked_ok":[]}
74832025-05-08T22:09:08.297ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 11.19585ms iteration = 22 status = {"errors":[],"garbage_collect_requested":[]}
74842025-05-08T22:09:08.297ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
74852025-05-08T22:09:08.297ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 27.228941ms iteration = 64 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
74862025-05-08T22:09:08.298ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
74872025-05-08T22:09:08.298ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
74882025-05-08T22:09:08.299ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
74892025-05-08T22:09:08.304ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
74902025-05-08T22:09:08.304ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
74912025-05-08T22:09:08.304ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
74922025-05-08T22:09:08.304ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
74932025-05-08T22:09:08.304ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
74942025-05-08T22:09:08.305ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
74952025-05-08T22:09:08.307ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 23.955906ms iteration = 64 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
74962025-05-08T22:09:08.312ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
74972025-05-08T22:09:08.312ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 28.833668ms iteration = 22 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
74982025-05-08T22:09:08.313ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
74992025-05-08T22:09:08.314ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1524, tv_nsec: 53195538 } })
75002025-05-08T22:09:08.315ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
75012025-05-08T22:09:08.315ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
75022025-05-08T22:09:08.316ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
75032025-05-08T22:09:08.320ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75042025-05-08T22:09:08.321ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75052025-05-08T22:09:08.323ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
75062025-05-08T22:09:08.323ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
75072025-05-08T22:09:08.324ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
75082025-05-08T22:09:08.325ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
75092025-05-08T22:09:08.325ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
75102025-05-08T22:09:08.326ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
75112025-05-08T22:09:08.327ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:552 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
75122025-05-08T22:09:08.327ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 43.523495ms iteration = 22 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
75132025-05-08T22:09:08.328ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
75142025-05-08T22:09:08.329ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
75152025-05-08T22:09:08.331ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75162025-05-08T22:09:08.334ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
75172025-05-08T22:09:08.334ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
75182025-05-08T22:09:08.334ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
75192025-05-08T22:09:08.344ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
75202025-05-08T22:09:08.345ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
75212025-05-08T22:09:08.346ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8d0bc72-bc41-47c9-b2a2-34d113085e95", "content-length": "26", "date": "Thu, 08 May 2025 22:09:08 GMT"} })
75222025-05-08T22:09:08.346ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
75232025-05-08T22:09:08.349ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
75242025-05-08T22:09:08.359ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
75252025-05-08T22:09:08.359ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
75262025-05-08T22:09:08.361ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
75272025-05-08T22:09:08.362ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
75282025-05-08T22:09:08.364ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75292025-05-08T22:09:08.365ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 79.096816ms iteration = 22 status = {"errors":[],"requests_created_ok":[]}
75302025-05-08T22:09:08.365ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 79.989795ms iteration = 22 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
75312025-05-08T22:09:08.368ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
75322025-05-08T22:09:08.368ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
75332025-05-08T22:09:08.368ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 2f2f2c32-8d58-4801-a669-9618eac0c3f7, lookup_type: ById(2f2f2c32-8d58-4801-a669-9618eac0c3f7) } result = Ok(())
75342025-05-08T22:09:08.368ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
75352025-05-08T22:09:08.369ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
75362025-05-08T22:09:08.369ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
75372025-05-08T22:09:08.369ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 85.388033ms iteration = 22 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
75382025-05-08T22:09:08.369ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
75392025-05-08T22:09:08.369ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
75402025-05-08T22:09:08.369ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
75412025-05-08T22:09:08.370ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 87.121882ms iteration = 22 status = 0
75422025-05-08T22:09:08.372ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
75432025-05-08T22:09:08.373ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
75442025-05-08T22:09:08.380ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
75452025-05-08T22:09:08.380ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
75462025-05-08T22:09:08.380ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "427fce24-f981-42ba-bf65-60a85f5b5500", "content-length": "26", "date": "Thu, 08 May 2025 22:09:08 GMT"} })
75472025-05-08T22:09:08.380ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
75482025-05-08T22:09:08.381ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
75492025-05-08T22:09:08.385ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
75502025-05-08T22:09:08.385ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
75512025-05-08T22:09:08.387ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75522025-05-08T22:09:08.387ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:362
75532025-05-08T22:09:08.387ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 114.97225ms iteration = 22 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
75542025-05-08T22:09:08.393ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
75552025-05-08T22:09:08.393ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
75562025-05-08T22:09:08.393ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
75572025-05-08T22:09:08.395ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75582025-05-08T22:09:08.397ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
75592025-05-08T22:09:08.397ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
75602025-05-08T22:09:08.397ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
75612025-05-08T22:09:08.404ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 118.955362ms iteration = 22 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
75622025-05-08T22:09:08.413ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e zone_address = fd00:1122:3344:101::2
75632025-05-08T22:09:08.413ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
75642025-05-08T22:09:08.414ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "a62123fe-7fa6-4ab2-b91f-f783844e0e1e", "content-length": "26", "date": "Thu, 08 May 2025 22:09:08 GMT"} })
75652025-05-08T22:09:08.414ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1174 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
75662025-05-08T22:09:08.415ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
75672025-05-08T22:09:08.458ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 189.986572ms iteration = 22 status = {}
75682025-05-08T22:09:08.463ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 179.982235ms iteration = 24 status = {"physical_disks_added":0}
75692025-05-08T22:09:08.563ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
75702025-05-08T22:09:08.563ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
75712025-05-08T22:09:08.564ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "451b5949-6760-40a8-b215-59de0ce2feff", "content-length": "372", "date": "Thu, 08 May 2025 22:09:08 GMT"} })
75722025-05-08T22:09:08.564ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
75732025-05-08T22:09:08.564ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
75742025-05-08T22:09:08.584ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
75752025-05-08T22:09:08.584ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
75762025-05-08T22:09:08.585ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "24332a5f-2a2c-49a6-b216-8a3f2b54c3f2", "date": "Thu, 08 May 2025 22:09:08 GMT"} })
75772025-05-08T22:09:08.585ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
75782025-05-08T22:09:08.585ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
75792025-05-08T22:09:08.592ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
75802025-05-08T22:09:08.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
75812025-05-08T22:09:08.597ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
75822025-05-08T22:09:08.597ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
75832025-05-08T22:09:08.601ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
75842025-05-08T22:09:08.601ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
75852025-05-08T22:09:08.602ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "aa9fbab6-c669-4e14-8717-ac21867baacb", "date": "Thu, 08 May 2025 22:09:08 GMT"} })
75862025-05-08T22:09:08.605ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 321.249926ms iteration = 22 status = {}
75872025-05-08T22:09:08.605ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
75882025-05-08T22:09:08.605ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
75892025-05-08T22:09:08.615ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
75902025-05-08T22:09:08.616ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "d95ffa99-aa10-4a4d-9649-142a8435862c", "content-length": "691", "date": "Thu, 08 May 2025 22:09:08 GMT"} })
75912025-05-08T22:09:08.616ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
75922025-05-08T22:09:08.616ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
75932025-05-08T22:09:08.616ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
75942025-05-08T22:09:08.617ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 333.962442ms iteration = 22 status = {}
75952025-05-08T22:09:08.617ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 349.505867ms iteration = 22 status = null
75962025-05-08T22:09:08.658ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
75972025-05-08T22:09:08.658ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
75982025-05-08T22:09:08.661ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: cd0b930c-534b-46ad-ac3b-2eaa94b0bf2c, rack_id: 64f3d6f4-89b8-48aa-a622-b684e261667e, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(65f83a36-18fb-49b7-9914-87c8517d200e) } rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
75992025-05-08T22:09:08.661ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
76002025-05-08T22:09:08.661ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
76012025-05-08T22:09:08.672ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1833 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
76022025-05-08T22:09:08.672ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1860 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 65f83a36-18fb-49b7-9914-87c8517d200e
76032025-05-08T22:09:08.672ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
76042025-05-08T22:09:08.672ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
76052025-05-08T22:09:08.672ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
76062025-05-08T22:09:08.672ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
76072025-05-08T22:09:08.672ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
76082025-05-08T22:09:08.672ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2007 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
76092025-05-08T22:09:08.672ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
76102025-05-08T22:09:08.672ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2043 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
76112025-05-08T22:09:08.672ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
76122025-05-08T22:09:08.672ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
76132025-05-08T22:09:08.673ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
76142025-05-08T22:09:08.675ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
76152025-05-08T22:09:08.675ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
76162025-05-08T22:09:08.844ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "677f3c0b-5bd4-42cb-8d8d-fe3ecef75070", "date": "Thu, 08 May 2025 22:09:08 GMT"} })
76172025-05-08T22:09:08.844ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch = Switch0
76182025-05-08T22:09:08.844ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
76192025-05-08T22:09:08.848ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0bda744-8f29-404f-ad1a-b919d3a3ab10", "content-length": "682", "date": "Thu, 08 May 2025 22:09:08 GMT"} })
76202025-05-08T22:09:08.848ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
76212025-05-08T22:09:08.848ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
76222025-05-08T22:09:08.852ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
76232025-05-08T22:09:08.852ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
76242025-05-08T22:09:08.856ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
76252025-05-08T22:09:08.856ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
76262025-05-08T22:09:08.858ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
76272025-05-08T22:09:08.858ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
76282025-05-08T22:09:08.865ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 581.613931ms iteration = 22 status = {}
76292025-05-08T22:09:11.938ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 271 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 659cdc34-ec8c-48a6-84b8-1d9c332c686d response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
76302025-05-08T22:09:18.270ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 65 reason = Timeout
76312025-05-08T22:09:18.270ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
76322025-05-08T22:09:18.271ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
76332025-05-08T22:09:18.272ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76342025-05-08T22:09:18.275ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
76352025-05-08T22:09:18.275ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
76362025-05-08T22:09:18.275ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
76372025-05-08T22:09:18.277ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
76382025-05-08T22:09:18.277ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
76392025-05-08T22:09:18.279ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76402025-05-08T22:09:18.282ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
76412025-05-08T22:09:18.282ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
76422025-05-08T22:09:18.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
76432025-05-08T22:09:18.283ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 65 reason = Timeout
76442025-05-08T22:09:18.283ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
76452025-05-08T22:09:18.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
76462025-05-08T22:09:18.285ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
76472025-05-08T22:09:18.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
76482025-05-08T22:09:18.286ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.190454ms iteration = 65 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
76492025-05-08T22:09:18.292ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
76502025-05-08T22:09:18.292ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.196827ms iteration = 65 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
76512025-05-08T22:09:21.939ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 277 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 3ba08b9c-7e69-44ba-a71a-4aa55fb48a9c response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
76522025-05-08T22:09:28.271ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 66 reason = Timeout
76532025-05-08T22:09:28.271ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
76542025-05-08T22:09:28.271ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
76552025-05-08T22:09:28.274ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76562025-05-08T22:09:28.277ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
76572025-05-08T22:09:28.277ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
76582025-05-08T22:09:28.278ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
76592025-05-08T22:09:28.280ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
76602025-05-08T22:09:28.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
76612025-05-08T22:09:28.283ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76622025-05-08T22:09:28.284ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 66 reason = Timeout
76632025-05-08T22:09:28.284ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
76642025-05-08T22:09:28.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
76652025-05-08T22:09:28.286ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
76662025-05-08T22:09:28.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
76672025-05-08T22:09:28.287ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
76682025-05-08T22:09:28.287ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
76692025-05-08T22:09:28.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
76702025-05-08T22:09:28.288ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.746161ms iteration = 66 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
76712025-05-08T22:09:28.301ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
76722025-05-08T22:09:28.301ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 30.513906ms iteration = 66 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
76732025-05-08T22:09:31.938ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 275 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = aa78b987-2937-4a73-aa2d-4703bdae49aa response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
76742025-05-08T22:09:36.931ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:43041 req_id = 3eb3afd4-98bf-4830-b70c-a32a2cf59f7b roles = RoleSet { roles: {} } uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?sort_by=id_ascending
76752025-05-08T22:09:36.932ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:43041 req_id = 3eb3afd4-98bf-4830-b70c-a32a2cf59f7b resource = Database result = Ok(()) uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?sort_by=id_ascending
76762025-05-08T22:09:36.934ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3233 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:43041 req_id = 3eb3afd4-98bf-4830-b70c-a32a2cf59f7b response_code = 200 uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?sort_by=id_ascending
76772025-05-08T22:09:36.935ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:43041 req_id = d1c64824-8b73-4081-ad1a-6840258c838b roles = RoleSet { roles: {} } uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZGQ4MjYzNC0yODQ0LTRmOTctOTc0Ni0wNzA5OTA0NGMxOWQifX0%3D
76782025-05-08T22:09:36.935ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:43041 req_id = d1c64824-8b73-4081-ad1a-6840258c838b resource = Database result = Ok(()) uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZGQ4MjYzNC0yODQ0LTRmOTctOTc0Ni0wNzA5OTA0NGMxOWQifX0%3D
76792025-05-08T22:09:36.937ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2148 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:43041 req_id = d1c64824-8b73-4081-ad1a-6840258c838b response_code = 200 uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZGQ4MjYzNC0yODQ0LTRmOTctOTc0Ni0wNzA5OTA0NGMxOWQifX0%3D
76802025-05-08T22:09:38.269ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 12 reason = Timeout
76812025-05-08T22:09:38.269ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
76822025-05-08T22:09:38.269ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 12 reason = Timeout
76832025-05-08T22:09:38.269ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
76842025-05-08T22:09:38.269ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
76852025-05-08T22:09:38.269ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 12 reason = Timeout
76862025-05-08T22:09:38.269ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 23 reason = Timeout
76872025-05-08T22:09:38.269ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
76882025-05-08T22:09:38.269ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
76892025-05-08T22:09:38.269ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
76902025-05-08T22:09:38.269ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 14 reason = Timeout
76912025-05-08T22:09:38.269ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 12 reason = Timeout
76922025-05-08T22:09:38.269ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-05-08 21:59:38.269709896 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
76932025-05-08T22:09:38.269ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
76942025-05-08T22:09:38.269ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 12 reason = Timeout
76952025-05-08T22:09:38.269ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 14 reason = Timeout
76962025-05-08T22:09:38.269ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 23 reason = Timeout
76972025-05-08T22:09:38.270ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
76982025-05-08T22:09:38.270ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 12 reason = Timeout
76992025-05-08T22:09:38.270ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
77002025-05-08T22:09:38.270ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
77012025-05-08T22:09:38.270ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
77022025-05-08T22:09:38.270ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
77032025-05-08T22:09:38.270ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
77042025-05-08T22:09:38.270ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
77052025-05-08T22:09:38.270ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("0ccabf41-81cd-45e7-bb26-335edaa2ee47.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("38a08dd6-6a75-4558-b4f8-df853f4d03d9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("38a08dd6-6a75-4558-b4f8-df853f4d03d9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::9))) }], valid_until: Instant { tv_sec: 1554, tv_nsec: 8653041 } })
77062025-05-08T22:09:38.270ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("228ed1c1-3cc1-4055-aae9-5595ad856b86.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("8b245480-f91d-406b-bb08-850e1c3e3fc5.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c8f20f67-ea63-4dbc-b0f1-c30876daff82.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c8f20f67-ea63-4dbc-b0f1-c30876daff82.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 1554, tv_nsec: 8645571 } })
77072025-05-08T22:09:38.270ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 67 reason = Timeout
77082025-05-08T22:09:38.270ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 13 reason = Timeout
77092025-05-08T22:09:38.270ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
77102025-05-08T22:09:38.270ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 23 reason = Timeout
77112025-05-08T22:09:38.270ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:85 reason = blueprint disabled target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2
77122025-05-08T22:09:38.270ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 11.99\u{b5}s iteration = 13 status = {"enabled":false,"target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2"}
77132025-05-08T22:09:38.270ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
77142025-05-08T22:09:38.271ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } dns_group = external
77152025-05-08T22:09:38.271ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.275784ms iteration = 12 status = {"addresses":["[fd00:1122:3344:101::8]:5353","[fd00:1122:3344:101::9]:5353"]}
77162025-05-08T22:09:38.271ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
77172025-05-08T22:09:38.271ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.340704ms iteration = 12 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:1::1]:5353"]}
77182025-05-08T22:09:38.272ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
77192025-05-08T22:09:38.272ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
77202025-05-08T22:09:38.272ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
77212025-05-08T22:09:38.272ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1554, tv_nsec: 10746924 } })
77222025-05-08T22:09:38.272ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 23 reason = Timeout
77232025-05-08T22:09:38.272ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
77242025-05-08T22:09:38.272ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
77252025-05-08T22:09:38.272ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1163
77262025-05-08T22:09:38.272ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77272025-05-08T22:09:38.272ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
77282025-05-08T22:09:38.274ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-05-08 21:59:38.269709896 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
77292025-05-08T22:09:38.274ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 4.991163ms iteration = 12 status = {"expiration":"2025-05-08T21:59:38.269709896Z","pruned":[]}
77302025-05-08T22:09:38.274ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
77312025-05-08T22:09:38.274ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
77322025-05-08T22:09:38.275ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
77332025-05-08T22:09:38.275ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1554, tv_nsec: 13288721 } })
77342025-05-08T22:09:38.275ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163
77352025-05-08T22:09:38.277ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77362025-05-08T22:09:38.277ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77372025-05-08T22:09:38.278ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77382025-05-08T22:09:38.278ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77392025-05-08T22:09:38.278ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 7.861624ms iteration = 23 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
77402025-05-08T22:09:38.281ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
77412025-05-08T22:09:38.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
77422025-05-08T22:09:38.283ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
77432025-05-08T22:09:38.283ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
77442025-05-08T22:09:38.283ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
77452025-05-08T22:09:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
77462025-05-08T22:09:38.284ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 67 reason = Timeout
77472025-05-08T22:09:38.284ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
77482025-05-08T22:09:38.284ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 23 reason = Timeout
77492025-05-08T22:09:38.284ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
77502025-05-08T22:09:38.284ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 23 reason = Timeout
77512025-05-08T22:09:38.284ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
77522025-05-08T22:09:38.284ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
77532025-05-08T22:09:38.284ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
77542025-05-08T22:09:38.284ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 23 reason = Timeout
77552025-05-08T22:09:38.284ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
77562025-05-08T22:09:38.284ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 23 reason = Timeout
77572025-05-08T22:09:38.284ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
77582025-05-08T22:09:38.284ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 25 reason = Timeout
77592025-05-08T22:09:38.284ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
77602025-05-08T22:09:38.284ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 12 reason = Timeout
77612025-05-08T22:09:38.284ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
77622025-05-08T22:09:38.284ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 23 reason = Timeout
77632025-05-08T22:09:38.284ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
77642025-05-08T22:09:38.284ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 23 reason = Timeout
77652025-05-08T22:09:38.284ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 12 reason = Timeout
77662025-05-08T22:09:38.284ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:53
77672025-05-08T22:09:38.284ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
77682025-05-08T22:09:38.284ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
77692025-05-08T22:09:38.284ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 23 reason = Timeout
77702025-05-08T22:09:38.284ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
77712025-05-08T22:09:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
77722025-05-08T22:09:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
77732025-05-08T22:09:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
77742025-05-08T22:09:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
77752025-05-08T22:09:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
77762025-05-08T22:09:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
77772025-05-08T22:09:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
77782025-05-08T22:09:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
77792025-05-08T22:09:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
77802025-05-08T22:09:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
77812025-05-08T22:09:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
77822025-05-08T22:09:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
77832025-05-08T22:09:38.285ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
77842025-05-08T22:09:38.285ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
77852025-05-08T22:09:38.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
77862025-05-08T22:09:38.286ZDEBGnexus (BackgroundTasks): activating background_task = webhook_deliverator iteration = 12 reason = Timeout
77872025-05-08T22:09:38.286ZDEBGnexus (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} }
77882025-05-08T22:09:38.286ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 12 reason = Timeout
77892025-05-08T22:09:38.286ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
77902025-05-08T22:09:38.286ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 23 reason = Timeout
77912025-05-08T22:09:38.286ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
77922025-05-08T22:09:38.286ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 23 reason = Timeout
77932025-05-08T22:09:38.286ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
77942025-05-08T22:09:38.286ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 23 reason = Timeout
77952025-05-08T22:09:38.286ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
77962025-05-08T22:09:38.286ZDEBGnexus (BackgroundTasks): activating background_task = webhook_dispatcher iteration = 12 reason = Timeout
77972025-05-08T22:09:38.286ZDEBGnexus (BackgroundTasks): roles background_task = webhook_dispatcher roles = RoleSet { roles: {} }
77982025-05-08T22:09:38.286ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 12 reason = Timeout
77992025-05-08T22:09:38.286ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
78002025-05-08T22:09:38.286ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 23 reason = Timeout
78012025-05-08T22:09:38.286ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
78022025-05-08T22:09:38.286ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 23 reason = Timeout
78032025-05-08T22:09:38.286ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
78042025-05-08T22:09:38.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
78052025-05-08T22:09:38.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database result = Ok(())
78062025-05-08T22:09:38.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
78072025-05-08T22:09:38.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
78082025-05-08T22:09:38.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
78092025-05-08T22:09:38.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
78102025-05-08T22:09:38.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
78112025-05-08T22:09:38.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database result = Ok(())
78122025-05-08T22:09:38.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
78132025-05-08T22:09:38.288ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78142025-05-08T22:09:38.289ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78152025-05-08T22:09:38.289ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
78162025-05-08T22:09:38.289ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
78172025-05-08T22:09:38.292ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
78182025-05-08T22:09:38.292ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
78192025-05-08T22:09:38.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
78202025-05-08T22:09:38.293ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
78212025-05-08T22:09:38.293ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
78222025-05-08T22:09:38.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
78232025-05-08T22:09:38.299ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78242025-05-08T22:09:38.300ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78252025-05-08T22:09:38.302ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
78262025-05-08T22:09:38.302ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
78272025-05-08T22:09:38.302ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78282025-05-08T22:09:38.302ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
78292025-05-08T22:09:38.305ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78302025-05-08T22:09:38.305ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
78312025-05-08T22:09:38.305ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
78322025-05-08T22:09:38.305ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
78332025-05-08T22:09:38.307ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
78342025-05-08T22:09:38.307ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
78352025-05-08T22:09:38.307ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
78362025-05-08T22:09:38.308ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78372025-05-08T22:09:38.309ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
78382025-05-08T22:09:38.309ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
78392025-05-08T22:09:38.309ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
78402025-05-08T22:09:38.310ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
78412025-05-08T22:09:38.310ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
78422025-05-08T22:09:38.310ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 24.385341ms iteration = 12 status = {"errors":[],"found_port_ok":[]}
78432025-05-08T22:09:38.311ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
78442025-05-08T22:09:38.311ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
78452025-05-08T22:09:38.312ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
78462025-05-08T22:09:38.323ZDEBGnexus (BackgroundTasks): activation complete background_task = webhook_deliverator elapsed = 37.162027ms iteration = 12 status = {"by_rx":{},"error":null}
78472025-05-08T22:09:38.335ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78482025-05-08T22:09:38.337ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
78492025-05-08T22:09:38.337ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
78502025-05-08T22:09:38.339ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
78512025-05-08T22:09:38.339ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
78522025-05-08T22:09:38.339ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
78532025-05-08T22:09:38.339ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
78542025-05-08T22:09:38.339ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
78552025-05-08T22:09:38.345ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:552 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
78562025-05-08T22:09:38.345ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 60.986422ms iteration = 23 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
78572025-05-08T22:09:38.346ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
78582025-05-08T22:09:38.346ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
78592025-05-08T22:09:38.348ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1554, tv_nsec: 86653285 } })
78602025-05-08T22:09:38.348ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
78612025-05-08T22:09:38.349ZDEBGnexus (BackgroundTasks): roles background_task = webhook_dispatcher roles = RoleSet { roles: {} }
78622025-05-08T22:09:38.349ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database result = Ok(())
78632025-05-08T22:09:38.349ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78642025-05-08T22:09:38.350ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78652025-05-08T22:09:38.353ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
78662025-05-08T22:09:38.353ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
78672025-05-08T22:09:38.353ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
78682025-05-08T22:09:38.354ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
78692025-05-08T22:09:38.354ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
78702025-05-08T22:09:38.354ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
78712025-05-08T22:09:38.354ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
78722025-05-08T22:09:38.354ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
78732025-05-08T22:09:38.354ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
78742025-05-08T22:09:38.354ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
78752025-05-08T22:09:38.354ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
78762025-05-08T22:09:38.354ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
78772025-05-08T22:09:38.354ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
78782025-05-08T22:09:38.354ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
78792025-05-08T22:09:38.354ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
78802025-05-08T22:09:38.354ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
78812025-05-08T22:09:38.359ZDEBGnexus (BackgroundTasks): activation complete background_task = webhook_dispatcher elapsed = 73.349714ms iteration = 12 status = {"dispatched":[],"errors":[],"glob_version":"140.0.0","globs_reprocessed":{},"no_receivers":[]}
78822025-05-08T22:09:38.363ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78832025-05-08T22:09:38.366ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
78842025-05-08T22:09:38.366ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
78852025-05-08T22:09:38.366ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
78862025-05-08T22:09:38.369ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78872025-05-08T22:09:38.370ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
78882025-05-08T22:09:38.370ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
78892025-05-08T22:09:38.370ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
78902025-05-08T22:09:38.370ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
78912025-05-08T22:09:38.373ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
78922025-05-08T22:09:38.373ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 102.658788ms iteration = 67 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
78932025-05-08T22:09:38.374ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
78942025-05-08T22:09:38.374ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
78952025-05-08T22:09:38.374ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
78962025-05-08T22:09:38.375ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 91.072454ms iteration = 25 status = {"physical_disks_added":0}
78972025-05-08T22:09:38.376ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
78982025-05-08T22:09:38.376ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
78992025-05-08T22:09:38.376ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
79002025-05-08T22:09:38.377ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
79012025-05-08T22:09:38.380ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79022025-05-08T22:09:38.382ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:362
79032025-05-08T22:09:38.382ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 109.472041ms iteration = 23 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
79042025-05-08T22:09:38.382ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
79052025-05-08T22:09:38.382ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
79062025-05-08T22:09:38.382ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 97.992938ms iteration = 23 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
79072025-05-08T22:09:38.384ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
79082025-05-08T22:09:38.384ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
79092025-05-08T22:09:38.385ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
79102025-05-08T22:09:38.386ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
79112025-05-08T22:09:38.386ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
79122025-05-08T22:09:38.386ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
79132025-05-08T22:09:38.387ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
79142025-05-08T22:09:38.387ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
79152025-05-08T22:09:38.387ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 7586992a-6fc5-4073-8962-728778f16aa5 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
79162025-05-08T22:09:38.387ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
79172025-05-08T22:09:38.387ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 101.073361ms iteration = 23 status = {"errors":[],"finish_invoked_ok":[]}
79182025-05-08T22:09:38.387ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
79192025-05-08T22:09:38.388ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
79202025-05-08T22:09:38.388ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
79212025-05-08T22:09:38.388ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 101.999471ms iteration = 12 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
79222025-05-08T22:09:38.388ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 104.529968ms iteration = 23 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
79232025-05-08T22:09:38.388ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
79242025-05-08T22:09:38.388ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 104.71151ms iteration = 67 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
79252025-05-08T22:09:38.389ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
79262025-05-08T22:09:38.389ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
79272025-05-08T22:09:38.389ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79282025-05-08T22:09:38.390ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
79292025-05-08T22:09:38.390ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 103.81307ms iteration = 23 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
79302025-05-08T22:09:38.390ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
79312025-05-08T22:09:38.390ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
79322025-05-08T22:09:38.392ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 106.488649ms iteration = 23 status = {"errors":[],"garbage_collect_requested":[]}
79332025-05-08T22:09:38.393ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
79342025-05-08T22:09:38.393ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
79352025-05-08T22:09:38.393ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
79362025-05-08T22:09:38.393ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 107.134536ms iteration = 23 status = {"errors":[],"requests_created_ok":[]}
79372025-05-08T22:09:38.395ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
79382025-05-08T22:09:38.395ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
79392025-05-08T22:09:38.396ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
79402025-05-08T22:09:38.396ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
79412025-05-08T22:09:38.397ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 7586992a-6fc5-4073-8962-728778f16aa5 dns_zone_name = control-plane.oxide.internal found_so_far = 55 version = 1
79422025-05-08T22:09:38.397ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
79432025-05-08T22:09:38.397ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-05-08 21:59:21.182334 UTC dns_group = internal generation = 1
79442025-05-08T22:09:38.397ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 127.646095ms iteration = 12 status = {"generation":1}
79452025-05-08T22:09:38.398ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79462025-05-08T22:09:38.398ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
79472025-05-08T22:09:38.398ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
79482025-05-08T22:09:38.398ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
79492025-05-08T22:09:38.399ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
79502025-05-08T22:09:38.401ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 2f2f2c32-8d58-4801-a669-9618eac0c3f7, lookup_type: ById(2f2f2c32-8d58-4801-a669-9618eac0c3f7) } result = Ok(())
79512025-05-08T22:09:38.401ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
79522025-05-08T22:09:38.401ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
79532025-05-08T22:09:38.402ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79542025-05-08T22:09:38.403ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79552025-05-08T22:09:38.404ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 120.382947ms iteration = 23 status = 0
79562025-05-08T22:09:38.406ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
79572025-05-08T22:09:38.406ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
79582025-05-08T22:09:38.406ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
79592025-05-08T22:09:38.406ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
79602025-05-08T22:09:38.406ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
79612025-05-08T22:09:38.406ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
79622025-05-08T22:09:38.408ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79632025-05-08T22:09:38.410ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
79642025-05-08T22:09:38.410ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:137 found = 0 not_ready_to_be_deleted = 0
79652025-05-08T22:09:38.410ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 126.21111ms iteration = 12 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
79662025-05-08T22:09:38.411ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
79672025-05-08T22:09:38.411ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
79682025-05-08T22:09:38.412ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
79692025-05-08T22:09:38.430ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 144.434604ms iteration = 23 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
79702025-05-08T22:09:38.434ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
79712025-05-08T22:09:38.434ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = d8128696-db5a-4b88-914a-69b28494e50e dns_zone_name = oxide.test found_so_far = 0 version = 2
79722025-05-08T22:09:38.434ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
79732025-05-08T22:09:38.434ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
79742025-05-08T22:09:38.465ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
79752025-05-08T22:09:38.465ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
79762025-05-08T22:09:38.465ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "a74b5af7-2e7f-41a1-813f-6a449e179341", "content-length": "26", "date": "Thu, 08 May 2025 22:09:38 GMT"} })
79772025-05-08T22:09:38.465ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
79782025-05-08T22:09:38.466ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-05-08T21:58:35.560911Z, time_modified: 2025-05-08T21:58:35.560911Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, db_silo: Silo { identity: SiloIdentity { id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-05-08T21:59:21.709488Z, time_modified: 2025-05-08T21:59:21.709488Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, db_silo: Silo { identity: SiloIdentity { id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-05-08T21:59:21.709488Z, time_modified: 2025-05-08T21:59:21.709488Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47" }] }) }
79792025-05-08T22:09:38.466ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 197.025287ms iteration = 12 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"7137ee58-6aa7-4199-a248-feadf3b8bafb","tls_certs":["37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47"]}},"default_endpoint":{"silo_id":"7137ee58-6aa7-4199-a248-feadf3b8bafb","tls_certs":["37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
79802025-05-08T22:09:38.467ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
79812025-05-08T22:09:38.469ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
79822025-05-08T22:09:38.469ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
79832025-05-08T22:09:38.484ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79842025-05-08T22:09:38.487ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
79852025-05-08T22:09:38.487ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
79862025-05-08T22:09:38.487ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
79872025-05-08T22:09:38.488ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = d8128696-db5a-4b88-914a-69b28494e50e dns_zone_name = oxide.test found_so_far = 1 version = 2
79882025-05-08T22:09:38.488ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
79892025-05-08T22:09:38.488ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-05-08 21:59:21.759368 UTC dns_group = external generation = 2
79902025-05-08T22:09:38.488ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 219.318236ms iteration = 12 status = {"generation":2}
79912025-05-08T22:09:38.597ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e zone_address = fd00:1122:3344:101::2
79922025-05-08T22:09:38.598ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
79932025-05-08T22:09:38.598ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb85e768-a429-4f7e-aa5e-de78f959acc3", "content-length": "26", "date": "Thu, 08 May 2025 22:09:38 GMT"} })
79942025-05-08T22:09:38.598ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1174 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
79952025-05-08T22:09:38.605ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
79962025-05-08T22:09:38.650ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
79972025-05-08T22:09:38.650ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
79982025-05-08T22:09:38.651ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "5213effe-45ca-4ec8-ac14-d9498cba1c88", "content-length": "26", "date": "Thu, 08 May 2025 22:09:38 GMT"} })
79992025-05-08T22:09:38.651ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
80002025-05-08T22:09:38.653ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
80012025-05-08T22:09:38.662ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
80022025-05-08T22:09:38.693ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
80032025-05-08T22:09:38.693ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1c37a80-7493-429f-9615-bf1f2bc39eb4", "content-length": "691", "date": "Thu, 08 May 2025 22:09:38 GMT"} })
80042025-05-08T22:09:38.693ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
80052025-05-08T22:09:38.693ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
80062025-05-08T22:09:38.693ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
80072025-05-08T22:09:38.695ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 411.043017ms iteration = 23 status = {}
80082025-05-08T22:09:38.719ZINFOnexus (BackgroundTasks): instance reincarnation completed successfully background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:113 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0
80092025-05-08T22:09:38.719ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 435.366336ms iteration = 12 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
80102025-05-08T22:09:38.778ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
80112025-05-08T22:09:38.782ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
80122025-05-08T22:09:38.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
80132025-05-08T22:09:38.785ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 515.22245ms iteration = 23 status = null
80142025-05-08T22:09:38.789ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
80152025-05-08T22:09:38.789ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
80162025-05-08T22:09:38.789ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "05feacc6-e610-42d8-a259-6dc5faec763e", "content-length": "372", "date": "Thu, 08 May 2025 22:09:38 GMT"} })
80172025-05-08T22:09:38.789ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
80182025-05-08T22:09:38.790ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
80192025-05-08T22:09:38.805ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
80202025-05-08T22:09:38.805ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
80212025-05-08T22:09:38.805ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "14dcf6fd-7fe9-4a55-bd51-17ef6c179e26", "date": "Thu, 08 May 2025 22:09:38 GMT"} })
80222025-05-08T22:09:38.805ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
80232025-05-08T22:09:38.805ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
80242025-05-08T22:09:38.807ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
80252025-05-08T22:09:38.807ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
80262025-05-08T22:09:38.810ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
80272025-05-08T22:09:38.810ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
80282025-05-08T22:09:38.814ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
80292025-05-08T22:09:38.814ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
80302025-05-08T22:09:38.815ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "01e0f2d6-b5a6-49e1-9ae1-7014b26bd21f", "date": "Thu, 08 May 2025 22:09:38 GMT"} })
80312025-05-08T22:09:38.816ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 532.600177ms iteration = 23 status = {}
80322025-05-08T22:09:38.854ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 584.35006ms iteration = 23 status = {}
80332025-05-08T22:09:38.914ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
80342025-05-08T22:09:38.914ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "42c0c71d-8bfa-4cba-9470-8eb25b1aefb9", "date": "Thu, 08 May 2025 22:09:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
80352025-05-08T22:09:38.914ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
80362025-05-08T22:09:38.916ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "8e685b09-292d-443f-b106-23fd1dead7d3", "date": "Thu, 08 May 2025 22:09:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
80372025-05-08T22:09:38.916ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
80382025-05-08T22:09:38.918ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
80392025-05-08T22:09:38.918ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 648.439365ms iteration = 14 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
80402025-05-08T22:09:38.938ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
80412025-05-08T22:09:39.001ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
80422025-05-08T22:09:39.002ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
80432025-05-08T22:09:39.002ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
80442025-05-08T22:09:39.003ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "9614d5e6-04ee-4d84-bdc3-dfa0965274ff", "date": "Thu, 08 May 2025 22:09:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
80452025-05-08T22:09:39.003ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
80462025-05-08T22:09:39.004ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: cd0b930c-534b-46ad-ac3b-2eaa94b0bf2c, rack_id: 64f3d6f4-89b8-48aa-a622-b684e261667e, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(65f83a36-18fb-49b7-9914-87c8517d200e) } rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
80472025-05-08T22:09:39.004ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
80482025-05-08T22:09:39.004ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "298b8c84-75be-4486-9d42-6b2765db757a", "date": "Thu, 08 May 2025 22:09:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
80492025-05-08T22:09:39.004ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
80502025-05-08T22:09:39.004ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
80512025-05-08T22:09:39.006ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "32f0bcec-54f9-4daa-b023-15df509c6d6d", "date": "Thu, 08 May 2025 22:09:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
80522025-05-08T22:09:39.006ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
80532025-05-08T22:09:39.007ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
80542025-05-08T22:09:39.007ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 738.190376ms iteration = 14 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
80552025-05-08T22:09:39.017ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1833 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
80562025-05-08T22:09:39.017ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1860 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 65f83a36-18fb-49b7-9914-87c8517d200e
80572025-05-08T22:09:39.017ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
80582025-05-08T22:09:39.017ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
80592025-05-08T22:09:39.017ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
80602025-05-08T22:09:39.017ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
80612025-05-08T22:09:39.017ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
80622025-05-08T22:09:39.017ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2007 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
80632025-05-08T22:09:39.017ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
80642025-05-08T22:09:39.017ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2043 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
80652025-05-08T22:09:39.017ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
80662025-05-08T22:09:39.017ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
80672025-05-08T22:09:39.018ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
80682025-05-08T22:09:39.020ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
80692025-05-08T22:09:39.020ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
80702025-05-08T22:09:39.221ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "8f6882f6-c6d8-4740-a572-6dd01c7f665f", "date": "Thu, 08 May 2025 22:09:39 GMT"} })
80712025-05-08T22:09:39.221ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch = Switch0
80722025-05-08T22:09:39.221ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
80732025-05-08T22:09:39.221ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "ebf70370-b405-496e-be5d-57e4558dabb3", "content-length": "682", "date": "Thu, 08 May 2025 22:09:39 GMT"} })
80742025-05-08T22:09:39.222ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
80752025-05-08T22:09:39.222ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
80762025-05-08T22:09:39.225ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
80772025-05-08T22:09:39.225ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
80782025-05-08T22:09:39.228ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
80792025-05-08T22:09:39.229ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
80802025-05-08T22:09:39.230ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
80812025-05-08T22:09:39.231ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
80822025-05-08T22:09:39.237ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 953.100505ms iteration = 23 status = {}
80832025-05-08T22:09:39.267ZDEBGnexus (producer-registration-task): registering as oximeter producer with Nexus registration_count = 4
80842025-05-08T22:09:39.267ZDEBGnexus (producer-registration-task): will register with Nexus at [fd00:1122:3344:101::b]:12221
80852025-05-08T22:09:39.296ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::b]:12221/metrics/producers
80862025-05-08T22:09:39.296ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::b]:32848
80872025-05-08T22:09:39.296ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:32848 req_id = d75abd2a-2263-499c-b916-b37ad6745e8e roles = RoleSet { roles: {} } uri = /metrics/producers
80882025-05-08T22:09:39.296ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:32848 req_id = d75abd2a-2263-499c-b916-b37ad6745e8e resource = Database result = Ok(()) uri = /metrics/producers
80892025-05-08T22:09:39.337ZINFOnexus: assigned collector to new producer collector_id = 22ae79e2-3731-44c9-8346-78c5ddbfe577 file = nexus/src/app/oximeter.rs:76 producer_id = 85a39cf9-9419-458f-af2e-c8b40d8f5f79
80902025-05-08T22:09:39.338ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 41593 local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:32848 req_id = d75abd2a-2263-499c-b916-b37ad6745e8e response_code = 201 uri = /metrics/producers
80912025-05-08T22:09:39.338ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "d75abd2a-2263-499c-b916-b37ad6745e8e", "content-length": "41", "date": "Thu, 08 May 2025 22:09:39 GMT"} })
80922025-05-08T22:09:39.338ZDEBGnexus (producer-registration-task): successfully registered with Nexus, pausing until time to renew lease lease_duration = 600s wait_period = 150s
80932025-05-08T22:09:41.939ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 302 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 471c94fb-808d-4b4d-ba9e-a6fd455a331e response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
80942025-05-08T22:09:48.271ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 68 reason = Timeout
80952025-05-08T22:09:48.271ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
80962025-05-08T22:09:48.272ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
80972025-05-08T22:09:48.273ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80982025-05-08T22:09:48.276ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
80992025-05-08T22:09:48.276ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
81002025-05-08T22:09:48.276ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
81012025-05-08T22:09:48.278ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
81022025-05-08T22:09:48.279ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
81032025-05-08T22:09:48.280ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81042025-05-08T22:09:48.283ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
81052025-05-08T22:09:48.283ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
81062025-05-08T22:09:48.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
81072025-05-08T22:09:48.283ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 68 reason = Timeout
81082025-05-08T22:09:48.283ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
81092025-05-08T22:09:48.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
81102025-05-08T22:09:48.285ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
81112025-05-08T22:09:48.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
81122025-05-08T22:09:48.294ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
81132025-05-08T22:09:48.294ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.795404ms iteration = 68 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
81142025-05-08T22:09:48.318ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 34.980814ms iteration = 68 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
81152025-05-08T22:09:51.938ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 256 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = fb8ec557-f7b0-43aa-ad63-dbd00d1942e7 response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
81162025-05-08T22:09:58.270ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 69 reason = Timeout
81172025-05-08T22:09:58.270ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
81182025-05-08T22:09:58.271ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
81192025-05-08T22:09:58.272ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81202025-05-08T22:09:58.275ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
81212025-05-08T22:09:58.275ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
81222025-05-08T22:09:58.275ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
81232025-05-08T22:09:58.277ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
81242025-05-08T22:09:58.278ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
81252025-05-08T22:09:58.279ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81262025-05-08T22:09:58.282ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
81272025-05-08T22:09:58.282ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
81282025-05-08T22:09:58.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
81292025-05-08T22:09:58.284ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 69 reason = Timeout
81302025-05-08T22:09:58.284ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
81312025-05-08T22:09:58.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
81322025-05-08T22:09:58.286ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
81332025-05-08T22:09:58.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
81342025-05-08T22:09:58.290ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
81352025-05-08T22:09:58.290ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 19.369937ms iteration = 69 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
81362025-05-08T22:09:58.348ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 64.311078ms iteration = 69 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
81372025-05-08T22:10:01.939ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 289 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = e29d659b-4486-490a-881c-342cc786c96d response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
81382025-05-08T22:10:08.269ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 24 reason = Timeout
81392025-05-08T22:10:08.269ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
81402025-05-08T22:10:08.269ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 24 reason = Timeout
81412025-05-08T22:10:08.269ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
81422025-05-08T22:10:08.269ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
81432025-05-08T22:10:08.270ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
81442025-05-08T22:10:08.271ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 70 reason = Timeout
81452025-05-08T22:10:08.271ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 24 reason = Timeout
81462025-05-08T22:10:08.271ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
81472025-05-08T22:10:08.271ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
81482025-05-08T22:10:08.271ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1584, tv_nsec: 9683312 } })
81492025-05-08T22:10:08.272ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1163
81502025-05-08T22:10:08.272ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1584, tv_nsec: 10555562 } })
81512025-05-08T22:10:08.272ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 24 reason = Timeout
81522025-05-08T22:10:08.272ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
81532025-05-08T22:10:08.272ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163
81542025-05-08T22:10:08.273ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
81552025-05-08T22:10:08.273ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81562025-05-08T22:10:08.273ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 2.755639ms iteration = 24 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
81572025-05-08T22:10:08.274ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81582025-05-08T22:10:08.276ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
81592025-05-08T22:10:08.276ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
81602025-05-08T22:10:08.276ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
81612025-05-08T22:10:08.277ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
81622025-05-08T22:10:08.277ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
81632025-05-08T22:10:08.277ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
81642025-05-08T22:10:08.278ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
81652025-05-08T22:10:08.278ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
81662025-05-08T22:10:08.279ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81672025-05-08T22:10:08.282ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
81682025-05-08T22:10:08.282ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
81692025-05-08T22:10:08.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
81702025-05-08T22:10:08.284ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 70 reason = Timeout
81712025-05-08T22:10:08.284ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 24 reason = Timeout
81722025-05-08T22:10:08.284ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
81732025-05-08T22:10:08.284ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
81742025-05-08T22:10:08.284ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 24 reason = Timeout
81752025-05-08T22:10:08.284ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
81762025-05-08T22:10:08.284ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 24 reason = Timeout
81772025-05-08T22:10:08.284ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
81782025-05-08T22:10:08.284ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 24 reason = Timeout
81792025-05-08T22:10:08.284ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
81802025-05-08T22:10:08.284ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
81812025-05-08T22:10:08.284ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 24 reason = Timeout
81822025-05-08T22:10:08.284ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
81832025-05-08T22:10:08.284ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
81842025-05-08T22:10:08.284ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 24 reason = Timeout
81852025-05-08T22:10:08.284ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
81862025-05-08T22:10:08.284ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 26 reason = Timeout
81872025-05-08T22:10:08.284ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
81882025-05-08T22:10:08.284ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 24 reason = Timeout
81892025-05-08T22:10:08.285ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
81902025-05-08T22:10:08.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
81912025-05-08T22:10:08.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
81922025-05-08T22:10:08.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
81932025-05-08T22:10:08.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
81942025-05-08T22:10:08.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
81952025-05-08T22:10:08.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
81962025-05-08T22:10:08.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
81972025-05-08T22:10:08.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
81982025-05-08T22:10:08.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
81992025-05-08T22:10:08.286ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 24 reason = Timeout
82002025-05-08T22:10:08.286ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 24 reason = Timeout
82012025-05-08T22:10:08.286ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
82022025-05-08T22:10:08.286ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
82032025-05-08T22:10:08.286ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 24 reason = Timeout
82042025-05-08T22:10:08.286ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
82052025-05-08T22:10:08.286ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 24 reason = Timeout
82062025-05-08T22:10:08.286ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
82072025-05-08T22:10:08.286ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 24 reason = Timeout
82082025-05-08T22:10:08.286ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
82092025-05-08T22:10:08.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
82102025-05-08T22:10:08.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
82112025-05-08T22:10:08.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
82122025-05-08T22:10:08.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
82132025-05-08T22:10:08.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
82142025-05-08T22:10:08.286ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82152025-05-08T22:10:08.288ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82162025-05-08T22:10:08.290ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
82172025-05-08T22:10:08.290ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
82182025-05-08T22:10:08.290ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
82192025-05-08T22:10:08.291ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
82202025-05-08T22:10:08.291ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
82212025-05-08T22:10:08.291ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
82222025-05-08T22:10:08.295ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82232025-05-08T22:10:08.295ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
82242025-05-08T22:10:08.295ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 24.284889ms iteration = 70 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
82252025-05-08T22:10:08.296ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82262025-05-08T22:10:08.299ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
82272025-05-08T22:10:08.299ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
82282025-05-08T22:10:08.299ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
82292025-05-08T22:10:08.300ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
82302025-05-08T22:10:08.300ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
82312025-05-08T22:10:08.300ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
82322025-05-08T22:10:08.301ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 15.234093ms iteration = 24 status = {"errors":[],"finish_invoked_ok":[]}
82332025-05-08T22:10:08.301ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
82342025-05-08T22:10:08.301ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
82352025-05-08T22:10:08.303ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
82362025-05-08T22:10:08.304ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
82372025-05-08T22:10:08.305ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82382025-05-08T22:10:08.306ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
82392025-05-08T22:10:08.306ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
82402025-05-08T22:10:08.307ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 21.227727ms iteration = 24 status = {"errors":[],"requests_created_ok":[]}
82412025-05-08T22:10:08.308ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
82422025-05-08T22:10:08.308ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
82432025-05-08T22:10:08.308ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
82442025-05-08T22:10:08.331ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
82452025-05-08T22:10:08.331ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
82462025-05-08T22:10:08.334ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82472025-05-08T22:10:08.336ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82482025-05-08T22:10:08.337ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
82492025-05-08T22:10:08.337ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
82502025-05-08T22:10:08.338ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
82512025-05-08T22:10:08.339ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
82522025-05-08T22:10:08.339ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
82532025-05-08T22:10:08.340ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
82542025-05-08T22:10:08.343ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
82552025-05-08T22:10:08.343ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
82562025-05-08T22:10:08.343ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "91e4626c-60dc-4c5e-99bb-840f45b347cc", "content-length": "26", "date": "Thu, 08 May 2025 22:10:08 GMT"} })
82572025-05-08T22:10:08.343ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
82582025-05-08T22:10:08.346ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
82592025-05-08T22:10:08.351ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
82602025-05-08T22:10:08.352ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
82612025-05-08T22:10:08.353ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
82622025-05-08T22:10:08.353ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
82632025-05-08T22:10:08.354ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
82642025-05-08T22:10:08.354ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
82652025-05-08T22:10:08.357ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82662025-05-08T22:10:08.359ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
82672025-05-08T22:10:08.359ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
82682025-05-08T22:10:08.359ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
82692025-05-08T22:10:08.360ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
82702025-05-08T22:10:08.360ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
82712025-05-08T22:10:08.360ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
82722025-05-08T22:10:08.360ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
82732025-05-08T22:10:08.360ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
82742025-05-08T22:10:08.360ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
82752025-05-08T22:10:08.361ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 75.317246ms iteration = 24 status = {"errors":[],"garbage_collect_requested":[]}
82762025-05-08T22:10:08.361ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
82772025-05-08T22:10:08.362ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
82782025-05-08T22:10:08.365ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:552 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
82792025-05-08T22:10:08.365ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 81.045127ms iteration = 24 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
82802025-05-08T22:10:08.381ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:362
82812025-05-08T22:10:08.381ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 108.403779ms iteration = 24 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
82822025-05-08T22:10:08.396ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
82832025-05-08T22:10:08.396ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
82842025-05-08T22:10:08.396ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ad33901-d1bb-4653-9f6a-c5b787b24777", "content-length": "26", "date": "Thu, 08 May 2025 22:10:08 GMT"} })
82852025-05-08T22:10:08.396ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
82862025-05-08T22:10:08.397ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
82872025-05-08T22:10:08.435ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
82882025-05-08T22:10:08.436ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
82892025-05-08T22:10:08.445ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
82902025-05-08T22:10:08.445ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
82912025-05-08T22:10:08.445ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "81f20d2b-6fb1-45e6-a8ba-ce39369e6a19", "content-length": "372", "date": "Thu, 08 May 2025 22:10:08 GMT"} })
82922025-05-08T22:10:08.445ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
82932025-05-08T22:10:08.446ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
82942025-05-08T22:10:08.448ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1584, tv_nsec: 185913747 } })
82952025-05-08T22:10:08.448ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
82962025-05-08T22:10:08.450ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
82972025-05-08T22:10:08.450ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
82982025-05-08T22:10:08.451ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
82992025-05-08T22:10:08.451ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
83002025-05-08T22:10:08.451ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
83012025-05-08T22:10:08.451ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
83022025-05-08T22:10:08.456ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 186.665396ms iteration = 24 status = {}
83032025-05-08T22:10:08.467ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
83042025-05-08T22:10:08.467ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
83052025-05-08T22:10:08.469ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 199.429093ms iteration = 24 status = null
83062025-05-08T22:10:08.482ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83072025-05-08T22:10:08.482ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 196.29783ms iteration = 24 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
83082025-05-08T22:10:08.483ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
83092025-05-08T22:10:08.483ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
83102025-05-08T22:10:08.484ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83112025-05-08T22:10:08.484ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e zone_address = fd00:1122:3344:101::2
83122025-05-08T22:10:08.484ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
83132025-05-08T22:10:08.484ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
83142025-05-08T22:10:08.484ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
83152025-05-08T22:10:08.484ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
83162025-05-08T22:10:08.484ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "da8ae204-3039-46e2-8ac9-9bf840fa81a9", "content-length": "26", "date": "Thu, 08 May 2025 22:10:08 GMT"} })
83172025-05-08T22:10:08.484ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1174 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
83182025-05-08T22:10:08.485ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
83192025-05-08T22:10:08.486ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83202025-05-08T22:10:08.487ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 202.637447ms iteration = 24 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
83212025-05-08T22:10:08.487ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 202.91239ms iteration = 70 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
83222025-05-08T22:10:08.488ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 2f2f2c32-8d58-4801-a669-9618eac0c3f7, lookup_type: ById(2f2f2c32-8d58-4801-a669-9618eac0c3f7) } result = Ok(())
83232025-05-08T22:10:08.488ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
83242025-05-08T22:10:08.488ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
83252025-05-08T22:10:08.488ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
83262025-05-08T22:10:08.488ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
83272025-05-08T22:10:08.488ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
83282025-05-08T22:10:08.490ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 205.658239ms iteration = 24 status = 0
83292025-05-08T22:10:08.491ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
83302025-05-08T22:10:08.491ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
83312025-05-08T22:10:08.491ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 206.314837ms iteration = 24 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
83322025-05-08T22:10:08.500ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
83332025-05-08T22:10:08.500ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
83342025-05-08T22:10:08.501ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "b8b39cef-cf6c-4395-be00-345694b49b97", "date": "Thu, 08 May 2025 22:10:08 GMT"} })
83352025-05-08T22:10:08.501ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
83362025-05-08T22:10:08.501ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
83372025-05-08T22:10:08.504ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
83382025-05-08T22:10:08.504ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
83392025-05-08T22:10:08.507ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
83402025-05-08T22:10:08.507ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
83412025-05-08T22:10:08.511ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
83422025-05-08T22:10:08.511ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
83432025-05-08T22:10:08.511ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "3e3e348e-4694-4278-b0b5-d12afe2a3422", "date": "Thu, 08 May 2025 22:10:08 GMT"} })
83442025-05-08T22:10:08.513ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 228.376693ms iteration = 24 status = {}
83452025-05-08T22:10:08.548ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 262.157084ms iteration = 24 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
83462025-05-08T22:10:08.594ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 309.985245ms iteration = 26 status = {"physical_disks_added":0}
83472025-05-08T22:10:08.617ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
83482025-05-08T22:10:08.619ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
83492025-05-08T22:10:08.642ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
83502025-05-08T22:10:08.645ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: cd0b930c-534b-46ad-ac3b-2eaa94b0bf2c, rack_id: 64f3d6f4-89b8-48aa-a622-b684e261667e, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(65f83a36-18fb-49b7-9914-87c8517d200e) } rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
83512025-05-08T22:10:08.645ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
83522025-05-08T22:10:08.645ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
83532025-05-08T22:10:08.657ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1833 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
83542025-05-08T22:10:08.657ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1860 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 65f83a36-18fb-49b7-9914-87c8517d200e
83552025-05-08T22:10:08.658ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
83562025-05-08T22:10:08.658ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
83572025-05-08T22:10:08.658ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
83582025-05-08T22:10:08.658ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
83592025-05-08T22:10:08.658ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
83602025-05-08T22:10:08.658ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2007 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
83612025-05-08T22:10:08.658ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
83622025-05-08T22:10:08.658ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2043 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
83632025-05-08T22:10:08.658ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
83642025-05-08T22:10:08.658ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
83652025-05-08T22:10:08.658ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
83662025-05-08T22:10:08.660ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
83672025-05-08T22:10:08.660ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
83682025-05-08T22:10:08.741ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "57c9b4d7-1e03-4a42-a1ba-c0f9e51810b3", "content-length": "691", "date": "Thu, 08 May 2025 22:10:08 GMT"} })
83692025-05-08T22:10:08.741ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
83702025-05-08T22:10:08.741ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
83712025-05-08T22:10:08.741ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
83722025-05-08T22:10:08.743ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 458.366553ms iteration = 24 status = {}
83732025-05-08T22:10:08.850ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "d3fb263e-fd10-40fc-9dfb-eb90321d4c35", "date": "Thu, 08 May 2025 22:10:08 GMT"} })
83742025-05-08T22:10:08.850ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch = Switch0
83752025-05-08T22:10:08.850ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
83762025-05-08T22:10:08.851ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "be1f5164-7e74-44f3-a426-dc3b475122c0", "content-length": "682", "date": "Thu, 08 May 2025 22:10:08 GMT"} })
83772025-05-08T22:10:08.851ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
83782025-05-08T22:10:08.851ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
83792025-05-08T22:10:08.856ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
83802025-05-08T22:10:08.856ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
83812025-05-08T22:10:08.859ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
83822025-05-08T22:10:08.860ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
83832025-05-08T22:10:08.862ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
83842025-05-08T22:10:08.862ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
83852025-05-08T22:10:08.869ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 584.836406ms iteration = 24 status = {}
83862025-05-08T22:10:11.939ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 246 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = ae4e8167-58da-4af7-a6fc-1fae44389917 response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
83872025-05-08T22:10:18.271ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 71 reason = Timeout
83882025-05-08T22:10:18.272ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
83892025-05-08T22:10:18.272ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
83902025-05-08T22:10:18.273ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83912025-05-08T22:10:18.276ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
83922025-05-08T22:10:18.276ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
83932025-05-08T22:10:18.276ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
83942025-05-08T22:10:18.278ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
83952025-05-08T22:10:18.279ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
83962025-05-08T22:10:18.281ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83972025-05-08T22:10:18.284ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 71 reason = Timeout
83982025-05-08T22:10:18.284ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
83992025-05-08T22:10:18.284ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
84002025-05-08T22:10:18.284ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
84012025-05-08T22:10:18.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
84022025-05-08T22:10:18.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
84032025-05-08T22:10:18.297ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
84042025-05-08T22:10:18.297ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 25.337541ms iteration = 71 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
84052025-05-08T22:10:18.326ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
84062025-05-08T22:10:18.327ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
84072025-05-08T22:10:18.359ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 74.969652ms iteration = 71 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
84082025-05-08T22:10:21.939ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 267 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = be218043-234d-4659-a4b3-b8461d198dc2 response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
84092025-05-08T22:10:28.271ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 72 reason = Timeout
84102025-05-08T22:10:28.271ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
84112025-05-08T22:10:28.271ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
84122025-05-08T22:10:28.274ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84132025-05-08T22:10:28.276ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
84142025-05-08T22:10:28.276ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
84152025-05-08T22:10:28.276ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
84162025-05-08T22:10:28.278ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
84172025-05-08T22:10:28.278ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
84182025-05-08T22:10:28.279ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84192025-05-08T22:10:28.282ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
84202025-05-08T22:10:28.282ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
84212025-05-08T22:10:28.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
84222025-05-08T22:10:28.284ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 72 reason = Timeout
84232025-05-08T22:10:28.284ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
84242025-05-08T22:10:28.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
84252025-05-08T22:10:28.286ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
84262025-05-08T22:10:28.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
84272025-05-08T22:10:28.287ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.046723ms iteration = 72 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
84282025-05-08T22:10:28.291ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
84292025-05-08T22:10:28.291ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.345117ms iteration = 72 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
84302025-05-08T22:10:31.939ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 282 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 7c853fa3-10f1-40df-9357-acf5fbc53660 response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
84312025-05-08T22:10:38.269ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 25 reason = Timeout
84322025-05-08T22:10:38.269ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 15 reason = Timeout
84332025-05-08T22:10:38.269ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
84342025-05-08T22:10:38.269ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 13 reason = Timeout
84352025-05-08T22:10:38.269ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 13 reason = Timeout
84362025-05-08T22:10:38.269ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
84372025-05-08T22:10:38.269ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
84382025-05-08T22:10:38.269ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
84392025-05-08T22:10:38.269ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 25 reason = Timeout
84402025-05-08T22:10:38.269ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
84412025-05-08T22:10:38.269ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 13 reason = Timeout
84422025-05-08T22:10:38.269ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 13 reason = Timeout
84432025-05-08T22:10:38.269ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 13 reason = Timeout
84442025-05-08T22:10:38.269ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
84452025-05-08T22:10:38.269ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
84462025-05-08T22:10:38.269ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 15 reason = Timeout
84472025-05-08T22:10:38.269ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 13 reason = Timeout
84482025-05-08T22:10:38.269ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-05-08 22:00:38.269381827 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
84492025-05-08T22:10:38.269ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
84502025-05-08T22:10:38.269ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
84512025-05-08T22:10:38.269ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
84522025-05-08T22:10:38.269ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
84532025-05-08T22:10:38.269ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
84542025-05-08T22:10:38.269ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
84552025-05-08T22:10:38.269ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("0ccabf41-81cd-45e7-bb26-335edaa2ee47.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("38a08dd6-6a75-4558-b4f8-df853f4d03d9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("38a08dd6-6a75-4558-b4f8-df853f4d03d9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::9))) }], valid_until: Instant { tv_sec: 1614, tv_nsec: 7512669 } })
84562025-05-08T22:10:38.269ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
84572025-05-08T22:10:38.269ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("228ed1c1-3cc1-4055-aae9-5595ad856b86.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("8b245480-f91d-406b-bb08-850e1c3e3fc5.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c8f20f67-ea63-4dbc-b0f1-c30876daff82.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c8f20f67-ea63-4dbc-b0f1-c30876daff82.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 1614, tv_nsec: 7604620 } })
84582025-05-08T22:10:38.270ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } dns_group = external
84592025-05-08T22:10:38.270ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.077042ms iteration = 13 status = {"addresses":["[fd00:1122:3344:101::8]:5353","[fd00:1122:3344:101::9]:5353"]}
84602025-05-08T22:10:38.270ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
84612025-05-08T22:10:38.270ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.248103ms iteration = 13 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:1::1]:5353"]}
84622025-05-08T22:10:38.271ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
84632025-05-08T22:10:38.271ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
84642025-05-08T22:10:38.271ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 73 reason = Timeout
84652025-05-08T22:10:38.271ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 14 reason = Timeout
84662025-05-08T22:10:38.271ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:85 reason = blueprint disabled target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2
84672025-05-08T22:10:38.271ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 25 reason = Timeout
84682025-05-08T22:10:38.271ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 11.8\u{b5}s iteration = 14 status = {"enabled":false,"target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2"}
84692025-05-08T22:10:38.271ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
84702025-05-08T22:10:38.271ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
84712025-05-08T22:10:38.272ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1614, tv_nsec: 10200078 } })
84722025-05-08T22:10:38.272ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
84732025-05-08T22:10:38.272ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1163
84742025-05-08T22:10:38.272ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
84752025-05-08T22:10:38.273ZDEBGnexus (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 6 reason = Timeout
84762025-05-08T22:10:38.273ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
84772025-05-08T22:10:38.273ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 25 reason = Timeout
84782025-05-08T22:10:38.273ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
84792025-05-08T22:10:38.273ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
84802025-05-08T22:10:38.273ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
84812025-05-08T22:10:38.273ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84822025-05-08T22:10:38.273ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
84832025-05-08T22:10:38.273ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
84842025-05-08T22:10:38.273ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
84852025-05-08T22:10:38.273ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
84862025-05-08T22:10:38.273ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
84872025-05-08T22:10:38.273ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
84882025-05-08T22:10:38.273ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
84892025-05-08T22:10:38.274ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1614, tv_nsec: 11630303 } })
84902025-05-08T22:10:38.274ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
84912025-05-08T22:10:38.274ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163
84922025-05-08T22:10:38.274ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-05-08 22:00:38.269381827 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
84932025-05-08T22:10:38.274ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 5.496888ms iteration = 13 status = {"expiration":"2025-05-08T22:00:38.269381827Z","pruned":[]}
84942025-05-08T22:10:38.275ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
84952025-05-08T22:10:38.275ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
84962025-05-08T22:10:38.275ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
84972025-05-08T22:10:38.278ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84982025-05-08T22:10:38.278ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84992025-05-08T22:10:38.278ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85002025-05-08T22:10:38.279ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 7.443399ms iteration = 25 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
85012025-05-08T22:10:38.279ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85022025-05-08T22:10:38.281ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
85032025-05-08T22:10:38.281ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
85042025-05-08T22:10:38.281ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
85052025-05-08T22:10:38.281ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
85062025-05-08T22:10:38.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
85072025-05-08T22:10:38.281ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
85082025-05-08T22:10:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
85092025-05-08T22:10:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
85102025-05-08T22:10:38.283ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
85112025-05-08T22:10:38.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
85122025-05-08T22:10:38.283ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
85132025-05-08T22:10:38.283ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
85142025-05-08T22:10:38.283ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
85152025-05-08T22:10:38.283ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
85162025-05-08T22:10:38.283ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85172025-05-08T22:10:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
85182025-05-08T22:10:38.284ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 73 reason = Timeout
85192025-05-08T22:10:38.284ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
85202025-05-08T22:10:38.284ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 25 reason = Timeout
85212025-05-08T22:10:38.284ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
85222025-05-08T22:10:38.284ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 13 reason = Timeout
85232025-05-08T22:10:38.284ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 27 reason = Timeout
85242025-05-08T22:10:38.284ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:53
85252025-05-08T22:10:38.284ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 25 reason = Timeout
85262025-05-08T22:10:38.284ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 13 reason = Timeout
85272025-05-08T22:10:38.284ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
85282025-05-08T22:10:38.284ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
85292025-05-08T22:10:38.284ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
85302025-05-08T22:10:38.284ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
85312025-05-08T22:10:38.284ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 25 reason = Timeout
85322025-05-08T22:10:38.284ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
85332025-05-08T22:10:38.284ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
85342025-05-08T22:10:38.284ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 25 reason = Timeout
85352025-05-08T22:10:38.284ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
85362025-05-08T22:10:38.284ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
85372025-05-08T22:10:38.284ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 25 reason = Timeout
85382025-05-08T22:10:38.284ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
85392025-05-08T22:10:38.284ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 25 reason = Timeout
85402025-05-08T22:10:38.284ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
85412025-05-08T22:10:38.284ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 25 reason = Timeout
85422025-05-08T22:10:38.284ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
85432025-05-08T22:10:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
85442025-05-08T22:10:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
85452025-05-08T22:10:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
85462025-05-08T22:10:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
85472025-05-08T22:10:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
85482025-05-08T22:10:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
85492025-05-08T22:10:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
85502025-05-08T22:10:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
85512025-05-08T22:10:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
85522025-05-08T22:10:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
85532025-05-08T22:10:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
85542025-05-08T22:10:38.286ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 25 reason = Timeout
85552025-05-08T22:10:38.286ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
85562025-05-08T22:10:38.286ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 25 reason = Timeout
85572025-05-08T22:10:38.286ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
85582025-05-08T22:10:38.286ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
85592025-05-08T22:10:38.286ZDEBGnexus (BackgroundTasks): activating background_task = webhook_dispatcher iteration = 13 reason = Timeout
85602025-05-08T22:10:38.286ZDEBGnexus (BackgroundTasks): roles background_task = webhook_dispatcher roles = RoleSet { roles: {} }
85612025-05-08T22:10:38.286ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 13 reason = Timeout
85622025-05-08T22:10:38.286ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
85632025-05-08T22:10:38.286ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 13 reason = Timeout
85642025-05-08T22:10:38.286ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
85652025-05-08T22:10:38.286ZDEBGnexus (BackgroundTasks): activating background_task = webhook_deliverator iteration = 13 reason = Timeout
85662025-05-08T22:10:38.286ZDEBGnexus (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} }
85672025-05-08T22:10:38.286ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 25 reason = Timeout
85682025-05-08T22:10:38.286ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
85692025-05-08T22:10:38.286ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 25 reason = Timeout
85702025-05-08T22:10:38.286ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
85712025-05-08T22:10:38.286ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 25 reason = Timeout
85722025-05-08T22:10:38.286ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
85732025-05-08T22:10:38.286ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
85742025-05-08T22:10:38.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
85752025-05-08T22:10:38.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
85762025-05-08T22:10:38.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database result = Ok(())
85772025-05-08T22:10:38.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
85782025-05-08T22:10:38.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
85792025-05-08T22:10:38.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
85802025-05-08T22:10:38.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database result = Ok(())
85812025-05-08T22:10:38.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
85822025-05-08T22:10:38.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
85832025-05-08T22:10:38.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
85842025-05-08T22:10:38.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
85852025-05-08T22:10:38.287ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
85862025-05-08T22:10:38.287ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
85872025-05-08T22:10:38.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
85882025-05-08T22:10:38.287ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85892025-05-08T22:10:38.289ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85902025-05-08T22:10:38.290ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
85912025-05-08T22:10:38.290ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
85922025-05-08T22:10:38.290ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
85932025-05-08T22:10:38.290ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85942025-05-08T22:10:38.292ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
85952025-05-08T22:10:38.292ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
85962025-05-08T22:10:38.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
85972025-05-08T22:10:38.295ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
85982025-05-08T22:10:38.295ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
85992025-05-08T22:10:38.295ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
86002025-05-08T22:10:38.297ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86012025-05-08T22:10:38.297ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86022025-05-08T22:10:38.299ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86032025-05-08T22:10:38.301ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
86042025-05-08T22:10:38.301ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
86052025-05-08T22:10:38.301ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
86062025-05-08T22:10:38.302ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
86072025-05-08T22:10:38.302ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
86082025-05-08T22:10:38.302ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
86092025-05-08T22:10:38.302ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
86102025-05-08T22:10:38.302ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
86112025-05-08T22:10:38.303ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
86122025-05-08T22:10:38.313ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86132025-05-08T22:10:38.317ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
86142025-05-08T22:10:38.317ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
86152025-05-08T22:10:38.317ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
86162025-05-08T22:10:38.329ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86172025-05-08T22:10:38.329ZDEBGnexus (BackgroundTasks): activation complete background_task = webhook_deliverator elapsed = 43.04739ms iteration = 13 status = {"by_rx":{},"error":null}
86182025-05-08T22:10:38.332ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
86192025-05-08T22:10:38.332ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
86202025-05-08T22:10:38.334ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
86212025-05-08T22:10:38.334ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86222025-05-08T22:10:38.336ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 49.347168ms iteration = 13 status = {"errors":[],"found_port_ok":[]}
86232025-05-08T22:10:38.337ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86242025-05-08T22:10:38.338ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
86252025-05-08T22:10:38.338ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
86262025-05-08T22:10:38.339ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
86272025-05-08T22:10:38.342ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
86282025-05-08T22:10:38.342ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
86292025-05-08T22:10:38.342ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
86302025-05-08T22:10:38.351ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
86312025-05-08T22:10:38.352ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
86322025-05-08T22:10:38.362ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
86332025-05-08T22:10:38.362ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
86342025-05-08T22:10:38.363ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
86352025-05-08T22:10:38.364ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
86362025-05-08T22:10:38.367ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
86372025-05-08T22:10:38.367ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
86382025-05-08T22:10:38.375ZDEBGnexus (BackgroundTasks): roles background_task = webhook_dispatcher roles = RoleSet { roles: {} }
86392025-05-08T22:10:38.375ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database result = Ok(())
86402025-05-08T22:10:38.377ZDEBGnexus (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 104.643629ms iteration = 6 status = {"nsuccess":5}
86412025-05-08T22:10:38.382ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86422025-05-08T22:10:38.385ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
86432025-05-08T22:10:38.385ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
86442025-05-08T22:10:38.385ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
86452025-05-08T22:10:38.385ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
86462025-05-08T22:10:38.386ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
86472025-05-08T22:10:38.386ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
86482025-05-08T22:10:38.386ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
86492025-05-08T22:10:38.386ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
86502025-05-08T22:10:38.386ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
86512025-05-08T22:10:38.388ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86522025-05-08T22:10:38.391ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
86532025-05-08T22:10:38.391ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
86542025-05-08T22:10:38.392ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
86552025-05-08T22:10:38.392ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
86562025-05-08T22:10:38.392ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 121.278906ms iteration = 73 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
86572025-05-08T22:10:38.394ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86582025-05-08T22:10:38.395ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
86592025-05-08T22:10:38.396ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86602025-05-08T22:10:38.396ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
86612025-05-08T22:10:38.396ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86622025-05-08T22:10:38.396ZDEBGnexus (BackgroundTasks): activation complete background_task = webhook_dispatcher elapsed = 109.670033ms iteration = 13 status = {"dispatched":[],"errors":[],"glob_version":"140.0.0","globs_reprocessed":{},"no_receivers":[]}
86632025-05-08T22:10:38.397ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
86642025-05-08T22:10:38.397ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
86652025-05-08T22:10:38.397ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 111.177279ms iteration = 13 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
86662025-05-08T22:10:38.398ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
86672025-05-08T22:10:38.398ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
86682025-05-08T22:10:38.398ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
86692025-05-08T22:10:38.398ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
86702025-05-08T22:10:38.398ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
86712025-05-08T22:10:38.398ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
86722025-05-08T22:10:38.398ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
86732025-05-08T22:10:38.398ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
86742025-05-08T22:10:38.400ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
86752025-05-08T22:10:38.400ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
86762025-05-08T22:10:38.400ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
86772025-05-08T22:10:38.401ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
86782025-05-08T22:10:38.401ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
86792025-05-08T22:10:38.401ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
86802025-05-08T22:10:38.404ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
86812025-05-08T22:10:38.404ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
86822025-05-08T22:10:38.405ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86832025-05-08T22:10:38.406ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 119.67111ms iteration = 25 status = {"errors":[],"garbage_collect_requested":[]}
86842025-05-08T22:10:38.407ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1614, tv_nsec: 144696706 } })
86852025-05-08T22:10:38.407ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
86862025-05-08T22:10:38.408ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
86872025-05-08T22:10:38.408ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
86882025-05-08T22:10:38.411ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
86892025-05-08T22:10:38.411ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
86902025-05-08T22:10:38.411ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
86912025-05-08T22:10:38.411ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 124.709333ms iteration = 25 status = {"errors":[],"finish_invoked_ok":[]}
86922025-05-08T22:10:38.415ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 129.313233ms iteration = 25 status = {"errors":[],"requests_created_ok":[]}
86932025-05-08T22:10:38.417ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
86942025-05-08T22:10:38.417ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
86952025-05-08T22:10:38.418ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "5307e72f-8176-4e11-870d-5d5f829d80b3", "content-length": "26", "date": "Thu, 08 May 2025 22:10:38 GMT"} })
86962025-05-08T22:10:38.418ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
86972025-05-08T22:10:38.419ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
86982025-05-08T22:10:38.421ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
86992025-05-08T22:10:38.423ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
87002025-05-08T22:10:38.423ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
87012025-05-08T22:10:38.423ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 7586992a-6fc5-4073-8962-728778f16aa5 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
87022025-05-08T22:10:38.423ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
87032025-05-08T22:10:38.423ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
87042025-05-08T22:10:38.423ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
87052025-05-08T22:10:38.426ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87062025-05-08T22:10:38.429ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
87072025-05-08T22:10:38.429ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
87082025-05-08T22:10:38.429ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
87092025-05-08T22:10:38.440ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 156.769146ms iteration = 73 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
87102025-05-08T22:10:38.445ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
87112025-05-08T22:10:38.445ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:137 found = 0 not_ready_to_be_deleted = 0
87122025-05-08T22:10:38.445ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 161.68277ms iteration = 13 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
87132025-05-08T22:10:38.447ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:362
87142025-05-08T22:10:38.447ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 173.758038ms iteration = 25 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
87152025-05-08T22:10:38.455ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
87162025-05-08T22:10:38.455ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
87172025-05-08T22:10:38.455ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
87182025-05-08T22:10:38.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
87192025-05-08T22:10:38.461ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 177.019673ms iteration = 25 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
87202025-05-08T22:10:38.464ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
87212025-05-08T22:10:38.465ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
87222025-05-08T22:10:38.474ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
87232025-05-08T22:10:38.539ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
87242025-05-08T22:10:38.540ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
87252025-05-08T22:10:38.559ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
87262025-05-08T22:10:38.559ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
87272025-05-08T22:10:38.559ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f6aeb12-3510-49d3-ad16-f25db16824ed", "content-length": "26", "date": "Thu, 08 May 2025 22:10:38 GMT"} })
87282025-05-08T22:10:38.559ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
87292025-05-08T22:10:38.562ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
87302025-05-08T22:10:38.562ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 278.844643ms iteration = 27 status = {"physical_disks_added":0}
87312025-05-08T22:10:38.582ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = instance_watcher body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vmms/5474a63d-312e-4346-94cf-a31d2fc0fd33/state
87322025-05-08T22:10:38.583ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = instance_watcher result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vmms/5474a63d-312e-4346-94cf-a31d2fc0fd33/state", status: 200, headers: {"content-type": "application/json", "x-request-id": "654b445f-7d41-49d6-b3d9-1ba4f3f23c72", "content-length": "130", "date": "Thu, 08 May 2025 22:10:38 GMT"} })
87332025-05-08T22:10:38.591ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
87342025-05-08T22:10:38.591ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
87352025-05-08T22:10:38.592ZDEBGnexus (BackgroundTasks): updating instance state background_task = instance_watcher instance_id = e5d7468b-cff8-4a88-bdfd-7e5e4ba11a13 sled_id = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 state = SledVmmState { vmm_state: VmmRuntimeState { state: Running, gen: Generation(4), time_updated: 2025-05-08T22:10:31.765932366Z }, migration_in: None, migration_out: None } vmm_id = 5474a63d-312e-4346-94cf-a31d2fc0fd33
87362025-05-08T22:10:38.592ZINFOnexus (BackgroundTasks): received new VMM runtime state from sled agent background_task = instance_watcher file = nexus/src/app/instance.rs:2140 instance_id = e5d7468b-cff8-4a88-bdfd-7e5e4ba11a13 migration_state = Migrations { migration_in: None, migration_out: None } propolis_id = 5474a63d-312e-4346-94cf-a31d2fc0fd33 sled_id = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 vmm_id = 5474a63d-312e-4346-94cf-a31d2fc0fd33 vmm_state = VmmRuntimeState { state: Running, gen: Generation(4), time_updated: 2025-05-08T22:10:31.765932366Z }
87372025-05-08T22:10:38.592ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher instance_id = e5d7468b-cff8-4a88-bdfd-7e5e4ba11a13 roles = RoleSet { roles: {} } sled_id = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 vmm_id = 5474a63d-312e-4346-94cf-a31d2fc0fd33
87382025-05-08T22:10:38.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher instance_id = e5d7468b-cff8-4a88-bdfd-7e5e4ba11a13 resource = Database result = Ok(()) sled_id = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 vmm_id = 5474a63d-312e-4346-94cf-a31d2fc0fd33
87392025-05-08T22:10:38.593ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87402025-05-08T22:10:38.597ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
87412025-05-08T22:10:38.597ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
87422025-05-08T22:10:38.597ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
87432025-05-08T22:10:38.614ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
87442025-05-08T22:10:38.614ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "26c7b28e-0cc4-46ec-a49a-579651ba4c4d", "date": "Thu, 08 May 2025 22:10:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
87452025-05-08T22:10:38.614ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
87462025-05-08T22:10:38.616ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "cac4bc25-6664-4c57-ab77-4f1f48a3fa8f", "date": "Thu, 08 May 2025 22:10:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
87472025-05-08T22:10:38.616ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
87482025-05-08T22:10:38.618ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
87492025-05-08T22:10:38.618ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 349.134875ms iteration = 15 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
87502025-05-08T22:10:38.631ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
87512025-05-08T22:10:38.631ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
87522025-05-08T22:10:38.634ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
87532025-05-08T22:10:38.634ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
87542025-05-08T22:10:38.634ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 350.196395ms iteration = 25 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
87552025-05-08T22:10:38.638ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 351.622801ms iteration = 25 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
87562025-05-08T22:10:38.639ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
87572025-05-08T22:10:38.639ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = d8128696-db5a-4b88-914a-69b28494e50e dns_zone_name = oxide.test found_so_far = 0 version = 2
87582025-05-08T22:10:38.639ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
87592025-05-08T22:10:38.640ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
87602025-05-08T22:10:38.648ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 361.495596ms iteration = 25 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
87612025-05-08T22:10:38.648ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87622025-05-08T22:10:38.657ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
87632025-05-08T22:10:38.657ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
87642025-05-08T22:10:38.657ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87652025-05-08T22:10:38.657ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87662025-05-08T22:10:38.658ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
87672025-05-08T22:10:38.660ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
87682025-05-08T22:10:38.660ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
87692025-05-08T22:10:38.661ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
87702025-05-08T22:10:38.663ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 2f2f2c32-8d58-4801-a669-9618eac0c3f7, lookup_type: ById(2f2f2c32-8d58-4801-a669-9618eac0c3f7) } result = Ok(())
87712025-05-08T22:10:38.663ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
87722025-05-08T22:10:38.663ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
87732025-05-08T22:10:38.665ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 7586992a-6fc5-4073-8962-728778f16aa5 dns_zone_name = control-plane.oxide.internal found_so_far = 55 version = 1
87742025-05-08T22:10:38.665ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
87752025-05-08T22:10:38.665ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-05-08 21:59:21.182334 UTC dns_group = internal generation = 1
87762025-05-08T22:10:38.665ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 395.872474ms iteration = 13 status = {"generation":1}
87772025-05-08T22:10:38.666ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 382.450161ms iteration = 25 status = 0
87782025-05-08T22:10:38.670ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e zone_address = fd00:1122:3344:101::2
87792025-05-08T22:10:38.670ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
87802025-05-08T22:10:38.670ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "47f10f42-6a0d-4d63-b9d2-a22fe20ce3f0", "content-length": "26", "date": "Thu, 08 May 2025 22:10:38 GMT"} })
87812025-05-08T22:10:38.670ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1174 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
87822025-05-08T22:10:38.674ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
87832025-05-08T22:10:38.676ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:552 pruned_instances = 0 total_completed = 1 total_failed = 0 total_incomplete = 0 total_instances = 1 update_sagas_queued = 0
87842025-05-08T22:10:38.678ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 394.625501ms iteration = 25 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{"running":1},"pruned_instances":0,"total_instances":1,"update_sagas_queued":0}
87852025-05-08T22:10:38.687ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
87862025-05-08T22:10:38.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
87872025-05-08T22:10:38.691ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 421.824041ms iteration = 25 status = null
87882025-05-08T22:10:38.693ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
87892025-05-08T22:10:38.699ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87902025-05-08T22:10:38.702ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
87912025-05-08T22:10:38.702ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
87922025-05-08T22:10:38.703ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
87932025-05-08T22:10:38.703ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87942025-05-08T22:10:38.707ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
87952025-05-08T22:10:38.707ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
87962025-05-08T22:10:38.707ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
87972025-05-08T22:10:38.710ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = d8128696-db5a-4b88-914a-69b28494e50e dns_zone_name = oxide.test found_so_far = 1 version = 2
87982025-05-08T22:10:38.710ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
87992025-05-08T22:10:38.710ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-05-08 21:59:21.759368 UTC dns_group = external generation = 2
88002025-05-08T22:10:38.710ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 440.948516ms iteration = 13 status = {"generation":2}
88012025-05-08T22:10:38.731ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 462.141363ms iteration = 25 status = {}
88022025-05-08T22:10:38.757ZINFOnexus (BackgroundTasks): found Silo/DNS/TLS config (changed) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-05-08T21:58:35.560911Z, time_modified: 2025-05-08T21:58:35.560911Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, db_silo: Silo { identity: SiloIdentity { id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-05-08T21:59:21.709488Z, time_modified: 2025-05-08T21:59:21.709488Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, db_silo: Silo { identity: SiloIdentity { id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-05-08T21:59:21.709488Z, time_modified: 2025-05-08T21:59:21.709488Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47" }] }) } file = nexus/src/app/background/tasks/external_endpoints.rs:94
88032025-05-08T22:10:38.757ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 488.222912ms iteration = 13 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"7137ee58-6aa7-4199-a248-feadf3b8bafb","tls_certs":["37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47"]}},"default_endpoint":{"silo_id":"7137ee58-6aa7-4199-a248-feadf3b8bafb","tls_certs":["37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
88042025-05-08T22:10:38.821ZINFOnexus (BackgroundTasks): instance reincarnation completed successfully background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:113 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0
88052025-05-08T22:10:38.821ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 536.921843ms iteration = 13 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
88062025-05-08T22:10:38.900ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
88072025-05-08T22:10:38.900ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "22f09dc5-5ba8-440a-aa6e-dc2d2e641bcf", "content-length": "811", "date": "Thu, 08 May 2025 22:10:38 GMT"} })
88082025-05-08T22:10:38.900ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {VirtualNetworkInterfaceHost { physical_host_ip: fd00:1122:3344:101::1, virtual_ip: 172.30.0.5, virtual_mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), vni: Vni(14929187) }} sled = sock
88092025-05-08T22:10:38.900ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
88102025-05-08T22:10:38.900ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
88112025-05-08T22:10:38.902ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 618.591787ms iteration = 25 status = {}
88122025-05-08T22:10:38.912ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
88132025-05-08T22:10:38.913ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "0d495623-7755-4b29-ae3b-6302e423eebe", "date": "Thu, 08 May 2025 22:10:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
88142025-05-08T22:10:38.913ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
88152025-05-08T22:10:38.915ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "742efcf8-a32f-43c6-bc1d-6aeb67d7a35a", "date": "Thu, 08 May 2025 22:10:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
88162025-05-08T22:10:38.915ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
88172025-05-08T22:10:38.917ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "119e0141-1c9f-4ced-94dd-7da0a5475c18", "date": "Thu, 08 May 2025 22:10:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
88182025-05-08T22:10:38.917ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
88192025-05-08T22:10:38.920ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
88202025-05-08T22:10:38.920ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 651.771111ms iteration = 15 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
88212025-05-08T22:10:38.923ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
88222025-05-08T22:10:38.923ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
88232025-05-08T22:10:38.924ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "7cb1f193-940f-44f8-b177-48f07a25def8", "content-length": "585", "date": "Thu, 08 May 2025 22:10:38 GMT"} })
88242025-05-08T22:10:38.924ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
88252025-05-08T22:10:38.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
88262025-05-08T22:10:38.958ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
88272025-05-08T22:10:38.958ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
88282025-05-08T22:10:38.961ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: cd0b930c-534b-46ad-ac3b-2eaa94b0bf2c, rack_id: 64f3d6f4-89b8-48aa-a622-b684e261667e, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(65f83a36-18fb-49b7-9914-87c8517d200e) } rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
88292025-05-08T22:10:38.961ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
88302025-05-08T22:10:38.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
88312025-05-08T22:10:38.973ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1833 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
88322025-05-08T22:10:38.973ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1860 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 65f83a36-18fb-49b7-9914-87c8517d200e
88332025-05-08T22:10:38.974ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
88342025-05-08T22:10:38.974ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
88352025-05-08T22:10:38.974ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
88362025-05-08T22:10:38.974ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
88372025-05-08T22:10:38.974ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
88382025-05-08T22:10:38.974ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2007 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
88392025-05-08T22:10:38.974ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
88402025-05-08T22:10:38.974ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2043 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
88412025-05-08T22:10:38.974ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
88422025-05-08T22:10:38.974ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
88432025-05-08T22:10:38.974ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
88442025-05-08T22:10:38.977ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
88452025-05-08T22:10:38.977ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
88462025-05-08T22:10:38.982ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {3865b5a3-3731-4d4d-ad6e-4ea76d77d32c: {10.151.2.176: {fce29db4-4d60-4a8c-a381-3cb8175181b1}, 10.151.2.175: {fce29db4-4d60-4a8c-a381-3cb8175181b1}}} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
88472025-05-08T22:10:38.982ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
88482025-05-08T22:10:39.081ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "60c5655d-9b8c-47b1-8fd6-53d9ec49c890", "date": "Thu, 08 May 2025 22:10:38 GMT"} })
88492025-05-08T22:10:39.081ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
88502025-05-08T22:10:39.081ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
88512025-05-08T22:10:39.084ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
88522025-05-08T22:10:39.084ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
88532025-05-08T22:10:39.087ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
88542025-05-08T22:10:39.088ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
88552025-05-08T22:10:39.091ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
88562025-05-08T22:10:39.091ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
88572025-05-08T22:10:39.093ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
88582025-05-08T22:10:39.094ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
88592025-05-08T22:10:39.097ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
88602025-05-08T22:10:39.097ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
88612025-05-08T22:10:39.100ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
88622025-05-08T22:10:39.100ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
88632025-05-08T22:10:39.100ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
88642025-05-08T22:10:39.101ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "bc4898da-4a6b-4022-81e2-0952eed700b0", "date": "Thu, 08 May 2025 22:10:39 GMT"} })
88652025-05-08T22:10:39.103ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 819.560656ms iteration = 25 status = {}
88662025-05-08T22:10:39.196ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "b2a7d10f-838a-424a-a173-35e6a3c1c518", "date": "Thu, 08 May 2025 22:10:39 GMT"} })
88672025-05-08T22:10:39.196ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch = Switch0
88682025-05-08T22:10:39.196ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
88692025-05-08T22:10:39.196ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ae9e7b1-b562-43e5-a7b6-333db3f59577", "content-length": "682", "date": "Thu, 08 May 2025 22:10:39 GMT"} })
88702025-05-08T22:10:39.196ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
88712025-05-08T22:10:39.197ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
88722025-05-08T22:10:39.201ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
88732025-05-08T22:10:39.201ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
88742025-05-08T22:10:39.205ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
88752025-05-08T22:10:39.206ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
88762025-05-08T22:10:39.207ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
88772025-05-08T22:10:39.207ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
88782025-05-08T22:10:39.214ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 930.676335ms iteration = 25 status = {}
88792025-05-08T22:10:41.940ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 366 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 4ef96cde-f6fe-4067-9df4-5c11b1e0688a response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
88802025-05-08T22:10:48.271ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 74 reason = Timeout
88812025-05-08T22:10:48.271ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
88822025-05-08T22:10:48.271ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
88832025-05-08T22:10:48.273ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88842025-05-08T22:10:48.276ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
88852025-05-08T22:10:48.276ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
88862025-05-08T22:10:48.276ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
88872025-05-08T22:10:48.278ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
88882025-05-08T22:10:48.278ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
88892025-05-08T22:10:48.280ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88902025-05-08T22:10:48.283ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
88912025-05-08T22:10:48.283ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
88922025-05-08T22:10:48.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
88932025-05-08T22:10:48.285ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 74 reason = Timeout
88942025-05-08T22:10:48.285ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
88952025-05-08T22:10:48.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
88962025-05-08T22:10:48.296ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
88972025-05-08T22:10:48.296ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 24.705534ms iteration = 74 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
88982025-05-08T22:10:48.321ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
88992025-05-08T22:10:48.321ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
89002025-05-08T22:10:48.359ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 74.285115ms iteration = 74 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
89012025-05-08T22:10:51.932ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::d]:60426
89022025-05-08T22:10:51.932ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60426 req_id = 83558491-4cdd-48c0-8fff-9b49261bde86 roles = RoleSet { roles: {} } uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?sort_by=id_ascending
89032025-05-08T22:10:51.932ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60426 req_id = 83558491-4cdd-48c0-8fff-9b49261bde86 resource = Database result = Ok(()) uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?sort_by=id_ascending
89042025-05-08T22:10:51.935ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3105 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60426 req_id = 83558491-4cdd-48c0-8fff-9b49261bde86 response_code = 200 uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?sort_by=id_ascending
89052025-05-08T22:10:51.935ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60426 req_id = 087bbd54-a0d9-49b0-9923-77d928e43056 roles = RoleSet { roles: {} } uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZGQ4MjYzNC0yODQ0LTRmOTctOTc0Ni0wNzA5OTA0NGMxOWQifX0%3D
89062025-05-08T22:10:51.936ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60426 req_id = 087bbd54-a0d9-49b0-9923-77d928e43056 resource = Database result = Ok(()) uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZGQ4MjYzNC0yODQ0LTRmOTctOTc0Ni0wNzA5OTA0NGMxOWQifX0%3D
89072025-05-08T22:10:51.938ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2457 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60426 req_id = 087bbd54-a0d9-49b0-9923-77d928e43056 response_code = 200 uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZGQ4MjYzNC0yODQ0LTRmOTctOTc0Ni0wNzA5OTA0NGMxOWQifX0%3D
89082025-05-08T22:10:51.939ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 297 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 648b8397-4fa0-4d9c-bfea-7d58cda938b4 response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
89092025-05-08T22:10:58.271ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 75 reason = Timeout
89102025-05-08T22:10:58.271ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
89112025-05-08T22:10:58.272ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
89122025-05-08T22:10:58.273ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89132025-05-08T22:10:58.276ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
89142025-05-08T22:10:58.276ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
89152025-05-08T22:10:58.276ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
89162025-05-08T22:10:58.279ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
89172025-05-08T22:10:58.279ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
89182025-05-08T22:10:58.280ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89192025-05-08T22:10:58.283ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
89202025-05-08T22:10:58.283ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
89212025-05-08T22:10:58.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
89222025-05-08T22:10:58.284ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 75 reason = Timeout
89232025-05-08T22:10:58.284ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
89242025-05-08T22:10:58.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
89252025-05-08T22:10:58.294ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
89262025-05-08T22:10:58.294ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 23.009816ms iteration = 75 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
89272025-05-08T22:10:58.317ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
89282025-05-08T22:10:58.317ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
89292025-05-08T22:10:58.346ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 61.208224ms iteration = 75 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
89302025-05-08T22:11:01.940ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 463 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 4e2450f0-d14e-4e73-bd76-2f6702290d1e response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
89312025-05-08T22:11:06.932ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60426 req_id = df974143-8a5f-4974-9238-8582449ac154 roles = RoleSet { roles: {} } uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?sort_by=id_ascending
89322025-05-08T22:11:06.932ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60426 req_id = df974143-8a5f-4974-9238-8582449ac154 resource = Database result = Ok(()) uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?sort_by=id_ascending
89332025-05-08T22:11:06.935ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3511 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60426 req_id = df974143-8a5f-4974-9238-8582449ac154 response_code = 200 uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?sort_by=id_ascending
89342025-05-08T22:11:06.935ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60426 req_id = 38e9c509-d01a-4c6e-beda-fcc83d37b957 roles = RoleSet { roles: {} } uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZGQ4MjYzNC0yODQ0LTRmOTctOTc0Ni0wNzA5OTA0NGMxOWQifX0%3D
89352025-05-08T22:11:06.936ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60426 req_id = 38e9c509-d01a-4c6e-beda-fcc83d37b957 resource = Database result = Ok(()) uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZGQ4MjYzNC0yODQ0LTRmOTctOTc0Ni0wNzA5OTA0NGMxOWQifX0%3D
89362025-05-08T22:11:06.938ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2472 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60426 req_id = 38e9c509-d01a-4c6e-beda-fcc83d37b957 response_code = 200 uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZGQ4MjYzNC0yODQ0LTRmOTctOTc0Ni0wNzA5OTA0NGMxOWQifX0%3D
89372025-05-08T22:11:08.270ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 26 reason = Timeout
89382025-05-08T22:11:08.270ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
89392025-05-08T22:11:08.270ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 26 reason = Timeout
89402025-05-08T22:11:08.270ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
89412025-05-08T22:11:08.270ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
89422025-05-08T22:11:08.270ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
89432025-05-08T22:11:08.271ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 76 reason = Timeout
89442025-05-08T22:11:08.271ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 26 reason = Timeout
89452025-05-08T22:11:08.271ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
89462025-05-08T22:11:08.272ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
89472025-05-08T22:11:08.273ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1644, tv_nsec: 10607842 } })
89482025-05-08T22:11:08.273ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 26 reason = Timeout
89492025-05-08T22:11:08.273ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
89502025-05-08T22:11:08.273ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
89512025-05-08T22:11:08.273ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1644, tv_nsec: 10927096 } })
89522025-05-08T22:11:08.273ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1163
89532025-05-08T22:11:08.273ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163
89542025-05-08T22:11:08.273ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89552025-05-08T22:11:08.274ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89562025-05-08T22:11:08.275ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.75672ms iteration = 26 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
89572025-05-08T22:11:08.277ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
89582025-05-08T22:11:08.277ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
89592025-05-08T22:11:08.277ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
89602025-05-08T22:11:08.277ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
89612025-05-08T22:11:08.277ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
89622025-05-08T22:11:08.277ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
89632025-05-08T22:11:08.279ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
89642025-05-08T22:11:08.279ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
89652025-05-08T22:11:08.279ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
89662025-05-08T22:11:08.279ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
89672025-05-08T22:11:08.281ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89682025-05-08T22:11:08.283ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
89692025-05-08T22:11:08.283ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
89702025-05-08T22:11:08.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
89712025-05-08T22:11:08.284ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89722025-05-08T22:11:08.285ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 76 reason = Timeout
89732025-05-08T22:11:08.285ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
89742025-05-08T22:11:08.285ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 26 reason = Timeout
89752025-05-08T22:11:08.285ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 26 reason = Timeout
89762025-05-08T22:11:08.285ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
89772025-05-08T22:11:08.285ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
89782025-05-08T22:11:08.285ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
89792025-05-08T22:11:08.285ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 28 reason = Timeout
89802025-05-08T22:11:08.285ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
89812025-05-08T22:11:08.285ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 26 reason = Timeout
89822025-05-08T22:11:08.285ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
89832025-05-08T22:11:08.285ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 26 reason = Timeout
89842025-05-08T22:11:08.285ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 26 reason = Timeout
89852025-05-08T22:11:08.285ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
89862025-05-08T22:11:08.285ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
89872025-05-08T22:11:08.285ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 26 reason = Timeout
89882025-05-08T22:11:08.285ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
89892025-05-08T22:11:08.285ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
89902025-05-08T22:11:08.285ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 26 reason = Timeout
89912025-05-08T22:11:08.285ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
89922025-05-08T22:11:08.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
89932025-05-08T22:11:08.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
89942025-05-08T22:11:08.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
89952025-05-08T22:11:08.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
89962025-05-08T22:11:08.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
89972025-05-08T22:11:08.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
89982025-05-08T22:11:08.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
89992025-05-08T22:11:08.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
90002025-05-08T22:11:08.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
90012025-05-08T22:11:08.287ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 26 reason = Timeout
90022025-05-08T22:11:08.287ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
90032025-05-08T22:11:08.287ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 26 reason = Timeout
90042025-05-08T22:11:08.287ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
90052025-05-08T22:11:08.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
90062025-05-08T22:11:08.287ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 26 reason = Timeout
90072025-05-08T22:11:08.287ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
90082025-05-08T22:11:08.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
90092025-05-08T22:11:08.287ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 26 reason = Timeout
90102025-05-08T22:11:08.287ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
90112025-05-08T22:11:08.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
90122025-05-08T22:11:08.287ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 26 reason = Timeout
90132025-05-08T22:11:08.287ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
90142025-05-08T22:11:08.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
90152025-05-08T22:11:08.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
90162025-05-08T22:11:08.288ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
90172025-05-08T22:11:08.288ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
90182025-05-08T22:11:08.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
90192025-05-08T22:11:08.289ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90202025-05-08T22:11:08.291ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90212025-05-08T22:11:08.292ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
90222025-05-08T22:11:08.292ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
90232025-05-08T22:11:08.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
90242025-05-08T22:11:08.295ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90252025-05-08T22:11:08.295ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
90262025-05-08T22:11:08.295ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
90272025-05-08T22:11:08.295ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
90282025-05-08T22:11:08.298ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
90292025-05-08T22:11:08.298ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
90302025-05-08T22:11:08.298ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
90312025-05-08T22:11:08.304ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
90322025-05-08T22:11:08.305ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
90332025-05-08T22:11:08.306ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1644, tv_nsec: 44069950 } })
90342025-05-08T22:11:08.307ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
90352025-05-08T22:11:08.329ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
90362025-05-08T22:11:08.329ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 58.045431ms iteration = 76 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
90372025-05-08T22:11:08.349ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:362
90382025-05-08T22:11:08.349ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 76.096123ms iteration = 26 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
90392025-05-08T22:11:08.355ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90402025-05-08T22:11:08.356ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
90412025-05-08T22:11:08.356ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
90422025-05-08T22:11:08.356ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
90432025-05-08T22:11:08.357ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
90442025-05-08T22:11:08.357ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
90452025-05-08T22:11:08.358ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
90462025-05-08T22:11:08.359ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
90472025-05-08T22:11:08.359ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
90482025-05-08T22:11:08.359ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
90492025-05-08T22:11:08.362ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90502025-05-08T22:11:08.364ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
90512025-05-08T22:11:08.364ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
90522025-05-08T22:11:08.364ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
90532025-05-08T22:11:08.364ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
90542025-05-08T22:11:08.365ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90552025-05-08T22:11:08.365ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
90562025-05-08T22:11:08.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
90572025-05-08T22:11:08.366ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
90582025-05-08T22:11:08.366ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
90592025-05-08T22:11:08.366ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
90602025-05-08T22:11:08.368ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
90612025-05-08T22:11:08.368ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
90622025-05-08T22:11:08.369ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
90632025-05-08T22:11:08.370ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 84.492564ms iteration = 26 status = {"errors":[],"finish_invoked_ok":[]}
90642025-05-08T22:11:08.371ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90652025-05-08T22:11:08.371ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
90662025-05-08T22:11:08.371ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
90672025-05-08T22:11:08.374ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 89.303135ms iteration = 26 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
90682025-05-08T22:11:08.376ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
90692025-05-08T22:11:08.376ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
90702025-05-08T22:11:08.376ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 89.76433ms iteration = 26 status = {"errors":[],"requests_created_ok":[]}
90712025-05-08T22:11:08.376ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
90722025-05-08T22:11:08.377ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:552 pruned_instances = 1 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
90732025-05-08T22:11:08.377ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 92.009663ms iteration = 26 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":1,"total_instances":0,"update_sagas_queued":0}
90742025-05-08T22:11:08.378ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
90752025-05-08T22:11:08.379ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
90762025-05-08T22:11:08.380ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 95.262419ms iteration = 28 status = {"physical_disks_added":0}
90772025-05-08T22:11:08.382ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
90782025-05-08T22:11:08.382ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
90792025-05-08T22:11:08.388ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 101.189382ms iteration = 26 status = {"errors":[],"garbage_collect_requested":[]}
90802025-05-08T22:11:08.414ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 129.166571ms iteration = 76 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
90812025-05-08T22:11:08.417ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
90822025-05-08T22:11:08.417ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
90832025-05-08T22:11:08.420ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
90842025-05-08T22:11:08.420ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
90852025-05-08T22:11:08.420ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
90862025-05-08T22:11:08.420ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 135.365607ms iteration = 26 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
90872025-05-08T22:11:08.420ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
90882025-05-08T22:11:08.422ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90892025-05-08T22:11:08.425ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
90902025-05-08T22:11:08.426ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
90912025-05-08T22:11:08.427ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 2f2f2c32-8d58-4801-a669-9618eac0c3f7, lookup_type: ById(2f2f2c32-8d58-4801-a669-9618eac0c3f7) } result = Ok(())
90922025-05-08T22:11:08.427ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
90932025-05-08T22:11:08.427ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
90942025-05-08T22:11:08.429ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 144.143892ms iteration = 26 status = 0
90952025-05-08T22:11:08.433ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 146.210823ms iteration = 26 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
90962025-05-08T22:11:08.444ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e zone_address = fd00:1122:3344:101::2
90972025-05-08T22:11:08.444ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
90982025-05-08T22:11:08.445ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "2738c040-1ea8-4c5a-95ff-eba0a4f35134", "content-length": "26", "date": "Thu, 08 May 2025 22:11:08 GMT"} })
90992025-05-08T22:11:08.445ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1174 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
91002025-05-08T22:11:08.446ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
91012025-05-08T22:11:08.448ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
91022025-05-08T22:11:08.448ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
91032025-05-08T22:11:08.448ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b3bc702-19d9-4710-a79a-73a361894e94", "content-length": "26", "date": "Thu, 08 May 2025 22:11:08 GMT"} })
91042025-05-08T22:11:08.448ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
91052025-05-08T22:11:08.450ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
91062025-05-08T22:11:08.464ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
91072025-05-08T22:11:08.464ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
91082025-05-08T22:11:08.464ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b64b53a-7e6a-4ffa-a124-82378df6d25d", "content-length": "26", "date": "Thu, 08 May 2025 22:11:08 GMT"} })
91092025-05-08T22:11:08.465ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
91102025-05-08T22:11:08.465ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 178.154135ms iteration = 26 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
91112025-05-08T22:11:08.465ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
91122025-05-08T22:11:08.466ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
91132025-05-08T22:11:08.466ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
91142025-05-08T22:11:08.513ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
91152025-05-08T22:11:08.514ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
91162025-05-08T22:11:08.595ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
91172025-05-08T22:11:08.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
91182025-05-08T22:11:08.601ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
91192025-05-08T22:11:08.601ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
91202025-05-08T22:11:08.602ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "958dba30-d3e7-4aab-90c4-ce16e23927bf", "content-length": "372", "date": "Thu, 08 May 2025 22:11:08 GMT"} })
91212025-05-08T22:11:08.602ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
91222025-05-08T22:11:08.602ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
91232025-05-08T22:11:08.604ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 334.782071ms iteration = 26 status = null
91242025-05-08T22:11:08.636ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91252025-05-08T22:11:08.638ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
91262025-05-08T22:11:08.638ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
91272025-05-08T22:11:08.638ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
91282025-05-08T22:11:08.639ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91292025-05-08T22:11:08.641ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
91302025-05-08T22:11:08.641ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
91312025-05-08T22:11:08.641ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
91322025-05-08T22:11:08.654ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
91332025-05-08T22:11:08.654ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
91342025-05-08T22:11:08.654ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "391a590d-a87f-4894-aa0e-d95c50903ffc", "date": "Thu, 08 May 2025 22:11:08 GMT"} })
91352025-05-08T22:11:08.654ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
91362025-05-08T22:11:08.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
91372025-05-08T22:11:08.659ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
91382025-05-08T22:11:08.659ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
91392025-05-08T22:11:08.663ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
91402025-05-08T22:11:08.663ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
91412025-05-08T22:11:08.664ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 394.698191ms iteration = 26 status = {}
91422025-05-08T22:11:08.694ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
91432025-05-08T22:11:08.698ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
91442025-05-08T22:11:08.698ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
91452025-05-08T22:11:08.727ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
91462025-05-08T22:11:08.727ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
91472025-05-08T22:11:08.729ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: cd0b930c-534b-46ad-ac3b-2eaa94b0bf2c, rack_id: 64f3d6f4-89b8-48aa-a622-b684e261667e, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(65f83a36-18fb-49b7-9914-87c8517d200e) } rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
91482025-05-08T22:11:08.729ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
91492025-05-08T22:11:08.730ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
91502025-05-08T22:11:08.741ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1833 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
91512025-05-08T22:11:08.741ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1860 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 65f83a36-18fb-49b7-9914-87c8517d200e
91522025-05-08T22:11:08.741ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
91532025-05-08T22:11:08.741ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
91542025-05-08T22:11:08.741ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
91552025-05-08T22:11:08.741ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
91562025-05-08T22:11:08.741ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
91572025-05-08T22:11:08.741ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2007 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
91582025-05-08T22:11:08.741ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
91592025-05-08T22:11:08.741ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2043 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
91602025-05-08T22:11:08.741ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
91612025-05-08T22:11:08.741ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
91622025-05-08T22:11:08.742ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
91632025-05-08T22:11:08.744ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
91642025-05-08T22:11:08.744ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
91652025-05-08T22:11:08.775ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "c41ca2f6-97fc-4385-9708-773c5cb00c7b", "content-length": "691", "date": "Thu, 08 May 2025 22:11:08 GMT"} })
91662025-05-08T22:11:08.775ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
91672025-05-08T22:11:08.775ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
91682025-05-08T22:11:08.775ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
91692025-05-08T22:11:08.776ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 491.397815ms iteration = 26 status = {}
91702025-05-08T22:11:08.780ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "df61267b-7680-4b89-a1f1-a712ae853944", "date": "Thu, 08 May 2025 22:11:08 GMT"} })
91712025-05-08T22:11:08.782ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 497.336669ms iteration = 26 status = {}
91722025-05-08T22:11:08.875ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "dcdb4cda-f18e-48ee-93d2-93aa89ea502b", "date": "Thu, 08 May 2025 22:11:08 GMT"} })
91732025-05-08T22:11:08.875ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch = Switch0
91742025-05-08T22:11:08.875ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
91752025-05-08T22:11:08.875ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "52f53711-4f4b-49af-9a3f-b0c7cbd97cd4", "content-length": "682", "date": "Thu, 08 May 2025 22:11:08 GMT"} })
91762025-05-08T22:11:08.875ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
91772025-05-08T22:11:08.875ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
91782025-05-08T22:11:08.880ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
91792025-05-08T22:11:08.880ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
91802025-05-08T22:11:08.883ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
91812025-05-08T22:11:08.884ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
91822025-05-08T22:11:08.885ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
91832025-05-08T22:11:08.885ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
91842025-05-08T22:11:08.892ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 607.194664ms iteration = 26 status = {}
91852025-05-08T22:11:11.939ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 296 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 1d8ad4c0-4f22-4c35-bf7b-84959ee0dc7d response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
91862025-05-08T22:11:18.271ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 77 reason = Timeout
91872025-05-08T22:11:18.272ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
91882025-05-08T22:11:18.272ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
91892025-05-08T22:11:18.274ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91902025-05-08T22:11:18.276ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
91912025-05-08T22:11:18.276ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
91922025-05-08T22:11:18.277ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
91932025-05-08T22:11:18.279ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
91942025-05-08T22:11:18.279ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
91952025-05-08T22:11:18.281ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91962025-05-08T22:11:18.284ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
91972025-05-08T22:11:18.284ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
91982025-05-08T22:11:18.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
91992025-05-08T22:11:18.284ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 77 reason = Timeout
92002025-05-08T22:11:18.284ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
92012025-05-08T22:11:18.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
92022025-05-08T22:11:18.295ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
92032025-05-08T22:11:18.295ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 23.682543ms iteration = 77 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
92042025-05-08T22:11:18.319ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
92052025-05-08T22:11:18.320ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
92062025-05-08T22:11:18.321ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 37.169617ms iteration = 77 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
92072025-05-08T22:11:21.939ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 255 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = bf682bd4-01ec-4f78-bf80-c5e02804eca9 response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
92082025-05-08T22:11:28.272ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 78 reason = Timeout
92092025-05-08T22:11:28.272ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
92102025-05-08T22:11:28.272ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
92112025-05-08T22:11:28.274ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92122025-05-08T22:11:28.277ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
92132025-05-08T22:11:28.277ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
92142025-05-08T22:11:28.277ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
92152025-05-08T22:11:28.279ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
92162025-05-08T22:11:28.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
92172025-05-08T22:11:28.281ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92182025-05-08T22:11:28.284ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 78 reason = Timeout
92192025-05-08T22:11:28.284ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
92202025-05-08T22:11:28.284ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
92212025-05-08T22:11:28.284ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
92222025-05-08T22:11:28.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
92232025-05-08T22:11:28.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
92242025-05-08T22:11:28.286ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
92252025-05-08T22:11:28.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
92262025-05-08T22:11:28.288ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.397816ms iteration = 78 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
92272025-05-08T22:11:28.296ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
92282025-05-08T22:11:28.296ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 24.22749ms iteration = 78 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
92292025-05-08T22:11:31.939ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 248 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 1fa92842-c492-4d3b-9c19-aa8a3b20ad62 response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
92302025-05-08T22:11:36.932ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60426 req_id = d7623962-55c3-4a49-8bc7-ff549abd8bdc roles = RoleSet { roles: {} } uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?sort_by=id_ascending
92312025-05-08T22:11:36.933ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60426 req_id = d7623962-55c3-4a49-8bc7-ff549abd8bdc resource = Database result = Ok(()) uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?sort_by=id_ascending
92322025-05-08T22:11:36.935ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2985 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60426 req_id = d7623962-55c3-4a49-8bc7-ff549abd8bdc response_code = 200 uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?sort_by=id_ascending
92332025-05-08T22:11:36.935ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60426 req_id = 689a3087-03b0-4a31-9f51-ebd242037dd0 roles = RoleSet { roles: {} } uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZGQ4MjYzNC0yODQ0LTRmOTctOTc0Ni0wNzA5OTA0NGMxOWQifX0%3D
92342025-05-08T22:11:36.936ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60426 req_id = 689a3087-03b0-4a31-9f51-ebd242037dd0 resource = Database result = Ok(()) uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZGQ4MjYzNC0yODQ0LTRmOTctOTc0Ni0wNzA5OTA0NGMxOWQifX0%3D
92352025-05-08T22:11:36.938ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2941 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60426 req_id = 689a3087-03b0-4a31-9f51-ebd242037dd0 response_code = 200 uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZGQ4MjYzNC0yODQ0LTRmOTctOTc0Ni0wNzA5OTA0NGMxOWQifX0%3D
92362025-05-08T22:11:38.270ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 27 reason = Timeout
92372025-05-08T22:11:38.270ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
92382025-05-08T22:11:38.270ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 16 reason = Timeout
92392025-05-08T22:11:38.270ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 14 reason = Timeout
92402025-05-08T22:11:38.270ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
92412025-05-08T22:11:38.270ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
92422025-05-08T22:11:38.270ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 14 reason = Timeout
92432025-05-08T22:11:38.270ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 14 reason = Timeout
92442025-05-08T22:11:38.270ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
92452025-05-08T22:11:38.270ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 27 reason = Timeout
92462025-05-08T22:11:38.270ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
92472025-05-08T22:11:38.270ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
92482025-05-08T22:11:38.270ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 14 reason = Timeout
92492025-05-08T22:11:38.270ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
92502025-05-08T22:11:38.270ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 14 reason = Timeout
92512025-05-08T22:11:38.270ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-05-08 22:01:38.270316681 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
92522025-05-08T22:11:38.270ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 14 reason = Timeout
92532025-05-08T22:11:38.270ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
92542025-05-08T22:11:38.270ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 16 reason = Timeout
92552025-05-08T22:11:38.270ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
92562025-05-08T22:11:38.270ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
92572025-05-08T22:11:38.270ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("228ed1c1-3cc1-4055-aae9-5595ad856b86.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("8b245480-f91d-406b-bb08-850e1c3e3fc5.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c8f20f67-ea63-4dbc-b0f1-c30876daff82.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c8f20f67-ea63-4dbc-b0f1-c30876daff82.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 1674, tv_nsec: 7744201 } })
92582025-05-08T22:11:38.270ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
92592025-05-08T22:11:38.270ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
92602025-05-08T22:11:38.270ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
92612025-05-08T22:11:38.270ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
92622025-05-08T22:11:38.270ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("0ccabf41-81cd-45e7-bb26-335edaa2ee47.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("38a08dd6-6a75-4558-b4f8-df853f4d03d9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("38a08dd6-6a75-4558-b4f8-df853f4d03d9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::9))) }], valid_until: Instant { tv_sec: 1674, tv_nsec: 7939363 } })
92632025-05-08T22:11:38.271ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
92642025-05-08T22:11:38.271ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 956.44\u{b5}s iteration = 14 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:1::1]:5353"]}
92652025-05-08T22:11:38.271ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } dns_group = external
92662025-05-08T22:11:38.271ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 917.61\u{b5}s iteration = 14 status = {"addresses":["[fd00:1122:3344:101::8]:5353","[fd00:1122:3344:101::9]:5353"]}
92672025-05-08T22:11:38.271ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
92682025-05-08T22:11:38.271ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
92692025-05-08T22:11:38.271ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 79 reason = Timeout
92702025-05-08T22:11:38.271ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 15 reason = Timeout
92712025-05-08T22:11:38.271ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:85 reason = blueprint disabled target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2
92722025-05-08T22:11:38.271ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 27 reason = Timeout
92732025-05-08T22:11:38.271ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 9.03\u{b5}s iteration = 15 status = {"enabled":false,"target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2"}
92742025-05-08T22:11:38.272ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
92752025-05-08T22:11:38.272ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
92762025-05-08T22:11:38.272ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
92772025-05-08T22:11:38.273ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
92782025-05-08T22:11:38.273ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 27 reason = Timeout
92792025-05-08T22:11:38.273ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
92802025-05-08T22:11:38.273ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92812025-05-08T22:11:38.273ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
92822025-05-08T22:11:38.274ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1674, tv_nsec: 11367730 } })
92832025-05-08T22:11:38.274ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1163
92842025-05-08T22:11:38.274ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-05-08 22:01:38.270316681 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
92852025-05-08T22:11:38.274ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 4.524389ms iteration = 14 status = {"expiration":"2025-05-08T22:01:38.270316681Z","pruned":[]}
92862025-05-08T22:11:38.275ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1674, tv_nsec: 12459722 } })
92872025-05-08T22:11:38.275ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163
92882025-05-08T22:11:38.276ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
92892025-05-08T22:11:38.276ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
92902025-05-08T22:11:38.276ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
92912025-05-08T22:11:38.279ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92922025-05-08T22:11:38.279ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 7.982525ms iteration = 27 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
92932025-05-08T22:11:38.280ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
92942025-05-08T22:11:38.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
92952025-05-08T22:11:38.281ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92962025-05-08T22:11:38.283ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
92972025-05-08T22:11:38.285ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 79 reason = Timeout
92982025-05-08T22:11:38.285ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 27 reason = Timeout
92992025-05-08T22:11:38.285ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
93002025-05-08T22:11:38.285ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
93012025-05-08T22:11:38.285ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
93022025-05-08T22:11:38.285ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 27 reason = Timeout
93032025-05-08T22:11:38.285ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
93042025-05-08T22:11:38.285ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 29 reason = Timeout
93052025-05-08T22:11:38.285ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
93062025-05-08T22:11:38.285ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 14 reason = Timeout
93072025-05-08T22:11:38.285ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
93082025-05-08T22:11:38.285ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 27 reason = Timeout
93092025-05-08T22:11:38.285ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 27 reason = Timeout
93102025-05-08T22:11:38.285ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
93112025-05-08T22:11:38.285ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
93122025-05-08T22:11:38.285ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 27 reason = Timeout
93132025-05-08T22:11:38.285ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
93142025-05-08T22:11:38.285ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 14 reason = Timeout
93152025-05-08T22:11:38.285ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:53
93162025-05-08T22:11:38.285ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
93172025-05-08T22:11:38.285ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 27 reason = Timeout
93182025-05-08T22:11:38.285ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
93192025-05-08T22:11:38.285ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
93202025-05-08T22:11:38.285ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 27 reason = Timeout
93212025-05-08T22:11:38.285ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
93222025-05-08T22:11:38.285ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
93232025-05-08T22:11:38.285ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
93242025-05-08T22:11:38.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
93252025-05-08T22:11:38.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
93262025-05-08T22:11:38.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
93272025-05-08T22:11:38.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
93282025-05-08T22:11:38.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
93292025-05-08T22:11:38.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
93302025-05-08T22:11:38.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
93312025-05-08T22:11:38.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
93322025-05-08T22:11:38.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
93332025-05-08T22:11:38.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
93342025-05-08T22:11:38.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
93352025-05-08T22:11:38.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
93362025-05-08T22:11:38.287ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 27 reason = Timeout
93372025-05-08T22:11:38.287ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 27 reason = Timeout
93382025-05-08T22:11:38.287ZDEBGnexus (BackgroundTasks): activating background_task = webhook_deliverator iteration = 14 reason = Timeout
93392025-05-08T22:11:38.287ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
93402025-05-08T22:11:38.287ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
93412025-05-08T22:11:38.287ZDEBGnexus (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} }
93422025-05-08T22:11:38.287ZDEBGnexus (BackgroundTasks): activating background_task = webhook_dispatcher iteration = 14 reason = Timeout
93432025-05-08T22:11:38.287ZDEBGnexus (BackgroundTasks): roles background_task = webhook_dispatcher roles = RoleSet { roles: {} }
93442025-05-08T22:11:38.287ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 27 reason = Timeout
93452025-05-08T22:11:38.287ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
93462025-05-08T22:11:38.287ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 27 reason = Timeout
93472025-05-08T22:11:38.287ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 27 reason = Timeout
93482025-05-08T22:11:38.287ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
93492025-05-08T22:11:38.287ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
93502025-05-08T22:11:38.287ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 14 reason = Timeout
93512025-05-08T22:11:38.287ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
93522025-05-08T22:11:38.287ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 14 reason = Timeout
93532025-05-08T22:11:38.287ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
93542025-05-08T22:11:38.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
93552025-05-08T22:11:38.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database result = Ok(())
93562025-05-08T22:11:38.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
93572025-05-08T22:11:38.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database result = Ok(())
93582025-05-08T22:11:38.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
93592025-05-08T22:11:38.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
93602025-05-08T22:11:38.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
93612025-05-08T22:11:38.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
93622025-05-08T22:11:38.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
93632025-05-08T22:11:38.290ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
93642025-05-08T22:11:38.291ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
93652025-05-08T22:11:38.292ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93662025-05-08T22:11:38.294ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93672025-05-08T22:11:38.296ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
93682025-05-08T22:11:38.296ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
93692025-05-08T22:11:38.296ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
93702025-05-08T22:11:38.299ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
93712025-05-08T22:11:38.299ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
93722025-05-08T22:11:38.299ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
93732025-05-08T22:11:38.302ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93742025-05-08T22:11:38.303ZDEBGnexus (BackgroundTasks): activation complete background_task = webhook_deliverator elapsed = 16.186893ms iteration = 14 status = {"by_rx":{},"error":null}
93752025-05-08T22:11:38.308ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
93762025-05-08T22:11:38.308ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
93772025-05-08T22:11:38.308ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
93782025-05-08T22:11:38.309ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 21.50522ms iteration = 14 status = {"errors":[],"found_port_ok":[]}
93792025-05-08T22:11:38.341ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93802025-05-08T22:11:38.347ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
93812025-05-08T22:11:38.349ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93822025-05-08T22:11:38.350ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93832025-05-08T22:11:38.350ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93842025-05-08T22:11:38.352ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
93852025-05-08T22:11:38.352ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
93862025-05-08T22:11:38.352ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 64.747713ms iteration = 14 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
93872025-05-08T22:11:38.354ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
93882025-05-08T22:11:38.354ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
93892025-05-08T22:11:38.354ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
93902025-05-08T22:11:38.354ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
93912025-05-08T22:11:38.354ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
93922025-05-08T22:11:38.354ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
93932025-05-08T22:11:38.355ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
93942025-05-08T22:11:38.355ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
93952025-05-08T22:11:38.355ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
93962025-05-08T22:11:38.357ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
93972025-05-08T22:11:38.358ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
93982025-05-08T22:11:38.358ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
93992025-05-08T22:11:38.358ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
94002025-05-08T22:11:38.358ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
94012025-05-08T22:11:38.358ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
94022025-05-08T22:11:38.358ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
94032025-05-08T22:11:38.358ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
94042025-05-08T22:11:38.359ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94052025-05-08T22:11:38.360ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
94062025-05-08T22:11:38.361ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
94072025-05-08T22:11:38.361ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
94082025-05-08T22:11:38.361ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
94092025-05-08T22:11:38.362ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
94102025-05-08T22:11:38.362ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
94112025-05-08T22:11:38.362ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
94122025-05-08T22:11:38.363ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
94132025-05-08T22:11:38.363ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
94142025-05-08T22:11:38.363ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
94152025-05-08T22:11:38.363ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
94162025-05-08T22:11:38.363ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
94172025-05-08T22:11:38.364ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
94182025-05-08T22:11:38.364ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
94192025-05-08T22:11:38.365ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94202025-05-08T22:11:38.365ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94212025-05-08T22:11:38.366ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1674, tv_nsec: 103106291 } })
94222025-05-08T22:11:38.366ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
94232025-05-08T22:11:38.366ZDEBGnexus (BackgroundTasks): roles background_task = webhook_dispatcher roles = RoleSet { roles: {} }
94242025-05-08T22:11:38.366ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
94252025-05-08T22:11:38.366ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
94262025-05-08T22:11:38.366ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
94272025-05-08T22:11:38.366ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
94282025-05-08T22:11:38.367ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database result = Ok(())
94292025-05-08T22:11:38.367ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94302025-05-08T22:11:38.367ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
94312025-05-08T22:11:38.367ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
94322025-05-08T22:11:38.369ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
94332025-05-08T22:11:38.369ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
94342025-05-08T22:11:38.369ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
94352025-05-08T22:11:38.369ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
94362025-05-08T22:11:38.370ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
94372025-05-08T22:11:38.370ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
94382025-05-08T22:11:38.371ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
94392025-05-08T22:11:38.371ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
94402025-05-08T22:11:38.371ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
94412025-05-08T22:11:38.373ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94422025-05-08T22:11:38.375ZDEBGnexus (BackgroundTasks): activation complete background_task = webhook_dispatcher elapsed = 87.8455ms iteration = 14 status = {"dispatched":[],"errors":[],"glob_version":"140.0.0","globs_reprocessed":{},"no_receivers":[]}
94432025-05-08T22:11:38.378ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
94442025-05-08T22:11:38.378ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
94452025-05-08T22:11:38.378ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
94462025-05-08T22:11:38.380ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94472025-05-08T22:11:38.381ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:362
94482025-05-08T22:11:38.381ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 107.631821ms iteration = 27 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
94492025-05-08T22:11:38.382ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94502025-05-08T22:11:38.385ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
94512025-05-08T22:11:38.386ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
94522025-05-08T22:11:38.386ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
94532025-05-08T22:11:38.386ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
94542025-05-08T22:11:38.386ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
94552025-05-08T22:11:38.386ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
94562025-05-08T22:11:38.399ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 111.886507ms iteration = 27 status = {"errors":[],"garbage_collect_requested":[]}
94572025-05-08T22:11:38.400ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 112.667155ms iteration = 27 status = {"errors":[],"finish_invoked_ok":[]}
94582025-05-08T22:11:38.405ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
94592025-05-08T22:11:38.407ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
94602025-05-08T22:11:38.408ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
94612025-05-08T22:11:38.408ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
94622025-05-08T22:11:38.408ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
94632025-05-08T22:11:38.408ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
94642025-05-08T22:11:38.408ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
94652025-05-08T22:11:38.416ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 129.016419ms iteration = 27 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
94662025-05-08T22:11:38.419ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
94672025-05-08T22:11:38.420ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
94682025-05-08T22:11:38.422ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94692025-05-08T22:11:38.424ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 137.313269ms iteration = 27 status = {"errors":[],"requests_created_ok":[]}
94702025-05-08T22:11:38.426ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
94712025-05-08T22:11:38.427ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
94722025-05-08T22:11:38.427ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
94732025-05-08T22:11:38.427ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 155.310431ms iteration = 79 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
94742025-05-08T22:11:38.427ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
94752025-05-08T22:11:38.427ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
94762025-05-08T22:11:38.427ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
94772025-05-08T22:11:38.434ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
94782025-05-08T22:11:38.435ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
94792025-05-08T22:11:38.438ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
94802025-05-08T22:11:38.438ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
94812025-05-08T22:11:38.443ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94822025-05-08T22:11:38.446ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-05-08T21:58:35.560911Z, time_modified: 2025-05-08T21:58:35.560911Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, db_silo: Silo { identity: SiloIdentity { id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-05-08T21:59:21.709488Z, time_modified: 2025-05-08T21:59:21.709488Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, db_silo: Silo { identity: SiloIdentity { id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-05-08T21:59:21.709488Z, time_modified: 2025-05-08T21:59:21.709488Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47" }] }) }
94832025-05-08T22:11:38.446ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 176.172794ms iteration = 14 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"7137ee58-6aa7-4199-a248-feadf3b8bafb","tls_certs":["37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47"]}},"default_endpoint":{"silo_id":"7137ee58-6aa7-4199-a248-feadf3b8bafb","tls_certs":["37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
94842025-05-08T22:11:38.446ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
94852025-05-08T22:11:38.446ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
94862025-05-08T22:11:38.446ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 161.250175ms iteration = 27 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
94872025-05-08T22:11:38.446ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
94882025-05-08T22:11:38.446ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
94892025-05-08T22:11:38.447ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
94902025-05-08T22:11:38.452ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
94912025-05-08T22:11:38.452ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:137 found = 0 not_ready_to_be_deleted = 0
94922025-05-08T22:11:38.452ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 167.325549ms iteration = 14 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
94932025-05-08T22:11:38.459ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 174.392625ms iteration = 27 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
94942025-05-08T22:11:38.467ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 181.911996ms iteration = 79 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
94952025-05-08T22:11:38.476ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:552 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
94962025-05-08T22:11:38.476ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 190.76778ms iteration = 27 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
94972025-05-08T22:11:38.477ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
94982025-05-08T22:11:38.481ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
94992025-05-08T22:11:38.481ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = d8128696-db5a-4b88-914a-69b28494e50e dns_zone_name = oxide.test found_so_far = 0 version = 2
95002025-05-08T22:11:38.481ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
95012025-05-08T22:11:38.482ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
95022025-05-08T22:11:38.482ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
95032025-05-08T22:11:38.482ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 7586992a-6fc5-4073-8962-728778f16aa5 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
95042025-05-08T22:11:38.482ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
95052025-05-08T22:11:38.483ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
95062025-05-08T22:11:38.484ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95072025-05-08T22:11:38.484ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95082025-05-08T22:11:38.488ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
95092025-05-08T22:11:38.488ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
95102025-05-08T22:11:38.489ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
95112025-05-08T22:11:38.489ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
95122025-05-08T22:11:38.489ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
95132025-05-08T22:11:38.489ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
95142025-05-08T22:11:38.491ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 7586992a-6fc5-4073-8962-728778f16aa5 dns_zone_name = control-plane.oxide.internal found_so_far = 55 version = 1
95152025-05-08T22:11:38.491ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
95162025-05-08T22:11:38.491ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-05-08 21:59:21.182334 UTC dns_group = internal generation = 1
95172025-05-08T22:11:38.491ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 221.526399ms iteration = 14 status = {"generation":1}
95182025-05-08T22:11:38.492ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = d8128696-db5a-4b88-914a-69b28494e50e dns_zone_name = oxide.test found_so_far = 1 version = 2
95192025-05-08T22:11:38.492ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
95202025-05-08T22:11:38.492ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-05-08 21:59:21.759368 UTC dns_group = external generation = 2
95212025-05-08T22:11:38.492ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 222.361728ms iteration = 14 status = {"generation":2}
95222025-05-08T22:11:38.497ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 209.815394ms iteration = 27 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
95232025-05-08T22:11:38.502ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95242025-05-08T22:11:38.503ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
95252025-05-08T22:11:38.503ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
95262025-05-08T22:11:38.506ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 2f2f2c32-8d58-4801-a669-9618eac0c3f7, lookup_type: ById(2f2f2c32-8d58-4801-a669-9618eac0c3f7) } result = Ok(())
95272025-05-08T22:11:38.506ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
95282025-05-08T22:11:38.507ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
95292025-05-08T22:11:38.507ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
95302025-05-08T22:11:38.507ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
95312025-05-08T22:11:38.509ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 224.302769ms iteration = 27 status = 0
95322025-05-08T22:11:38.545ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
95332025-05-08T22:11:38.545ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
95342025-05-08T22:11:38.546ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "28e1c9bf-0fae-44fa-9a93-b9beec03f716", "content-length": "26", "date": "Thu, 08 May 2025 22:11:38 GMT"} })
95352025-05-08T22:11:38.546ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
95362025-05-08T22:11:38.549ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
95372025-05-08T22:11:38.563ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
95382025-05-08T22:11:38.572ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
95392025-05-08T22:11:38.572ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
95402025-05-08T22:11:38.572ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "c2a93c5f-c557-4d1d-8278-07f866d65938", "content-length": "26", "date": "Thu, 08 May 2025 22:11:38 GMT"} })
95412025-05-08T22:11:38.572ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
95422025-05-08T22:11:38.573ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e zone_address = fd00:1122:3344:101::2
95432025-05-08T22:11:38.573ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
95442025-05-08T22:11:38.573ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "dde8e249-9a57-45b2-af34-0f3806cbc39f", "content-length": "26", "date": "Thu, 08 May 2025 22:11:38 GMT"} })
95452025-05-08T22:11:38.573ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1174 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
95462025-05-08T22:11:38.575ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
95472025-05-08T22:11:38.576ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
95482025-05-08T22:11:38.632ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 346.683358ms iteration = 29 status = {"physical_disks_added":0}
95492025-05-08T22:11:38.675ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
95502025-05-08T22:11:38.675ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
95512025-05-08T22:11:38.675ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ab6dfc0-a1fe-41e8-b31e-6a7ec465096b", "content-length": "372", "date": "Thu, 08 May 2025 22:11:38 GMT"} })
95522025-05-08T22:11:38.675ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
95532025-05-08T22:11:38.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
95542025-05-08T22:11:38.693ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
95552025-05-08T22:11:38.694ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
95562025-05-08T22:11:38.696ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95572025-05-08T22:11:38.697ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
95582025-05-08T22:11:38.697ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
95592025-05-08T22:11:38.698ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "9e08f527-7045-481b-80f5-2034c146608c", "date": "Thu, 08 May 2025 22:11:38 GMT"} })
95602025-05-08T22:11:38.698ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
95612025-05-08T22:11:38.698ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
95622025-05-08T22:11:38.699ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
95632025-05-08T22:11:38.699ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
95642025-05-08T22:11:38.699ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
95652025-05-08T22:11:38.700ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95662025-05-08T22:11:38.701ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
95672025-05-08T22:11:38.701ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
95682025-05-08T22:11:38.703ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
95692025-05-08T22:11:38.703ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
95702025-05-08T22:11:38.703ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
95712025-05-08T22:11:38.707ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
95722025-05-08T22:11:38.707ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
95732025-05-08T22:11:38.711ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
95742025-05-08T22:11:38.711ZINFOnexus (BackgroundTasks): instance reincarnation completed successfully background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:113 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0
95752025-05-08T22:11:38.711ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 426.37209ms iteration = 14 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
95762025-05-08T22:11:38.711ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
95772025-05-08T22:11:38.712ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "1f8bd274-e2c0-4e65-8070-aa913f903327", "date": "Thu, 08 May 2025 22:11:38 GMT"} })
95782025-05-08T22:11:38.713ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 428.309231ms iteration = 27 status = {}
95792025-05-08T22:11:38.731ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
95802025-05-08T22:11:38.732ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
95812025-05-08T22:11:38.734ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 464.64588ms iteration = 27 status = null
95822025-05-08T22:11:38.735ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 465.470209ms iteration = 27 status = {}
95832025-05-08T22:11:38.751ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
95842025-05-08T22:11:38.752ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "6becf303-59a5-4bbc-8f4e-2775ce84d5fa", "date": "Thu, 08 May 2025 22:11:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
95852025-05-08T22:11:38.752ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
95862025-05-08T22:11:38.754ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "cfe20ee3-d241-4450-b57a-d531c4721c38", "date": "Thu, 08 May 2025 22:11:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
95872025-05-08T22:11:38.754ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
95882025-05-08T22:11:38.755ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
95892025-05-08T22:11:38.755ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 485.581744ms iteration = 16 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
95902025-05-08T22:11:38.777ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
95912025-05-08T22:11:38.880ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
95922025-05-08T22:11:38.922ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6540046-fafc-40f0-bfce-c76f9a05537b", "content-length": "691", "date": "Thu, 08 May 2025 22:11:38 GMT"} })
95932025-05-08T22:11:38.922ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
95942025-05-08T22:11:38.922ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
95952025-05-08T22:11:38.923ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
95962025-05-08T22:11:38.924ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 639.076186ms iteration = 27 status = {}
95972025-05-08T22:11:38.943ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
95982025-05-08T22:11:38.943ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "9460af2a-5f95-4fbe-a88a-604be9e75d10", "date": "Thu, 08 May 2025 22:11:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
95992025-05-08T22:11:38.943ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
96002025-05-08T22:11:38.945ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "a1b5e926-0b72-40d3-9f53-131deb00bd4e", "date": "Thu, 08 May 2025 22:11:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
96012025-05-08T22:11:38.945ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
96022025-05-08T22:11:38.947ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "734a4fe1-2a8b-4bc7-abfb-2b8c4f0f86d6", "date": "Thu, 08 May 2025 22:11:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
96032025-05-08T22:11:38.947ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
96042025-05-08T22:11:38.948ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
96052025-05-08T22:11:38.949ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 678.82147ms iteration = 16 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
96062025-05-08T22:11:38.972ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
96072025-05-08T22:11:38.972ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
96082025-05-08T22:11:38.974ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: cd0b930c-534b-46ad-ac3b-2eaa94b0bf2c, rack_id: 64f3d6f4-89b8-48aa-a622-b684e261667e, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(65f83a36-18fb-49b7-9914-87c8517d200e) } rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
96092025-05-08T22:11:38.974ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
96102025-05-08T22:11:38.974ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
96112025-05-08T22:11:38.984ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1833 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
96122025-05-08T22:11:38.984ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1860 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 65f83a36-18fb-49b7-9914-87c8517d200e
96132025-05-08T22:11:38.985ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
96142025-05-08T22:11:38.985ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
96152025-05-08T22:11:38.985ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
96162025-05-08T22:11:38.985ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
96172025-05-08T22:11:38.985ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
96182025-05-08T22:11:38.985ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2007 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
96192025-05-08T22:11:38.985ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
96202025-05-08T22:11:38.985ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2043 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
96212025-05-08T22:11:38.985ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
96222025-05-08T22:11:38.985ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
96232025-05-08T22:11:38.986ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
96242025-05-08T22:11:38.988ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
96252025-05-08T22:11:38.988ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
96262025-05-08T22:11:39.087ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "a6319be9-795d-4192-b30e-e345963aa55f", "date": "Thu, 08 May 2025 22:11:38 GMT"} })
96272025-05-08T22:11:39.087ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch = Switch0
96282025-05-08T22:11:39.087ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
96292025-05-08T22:11:39.088ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "96f2df69-9d5c-48f3-9aba-139e6228ab77", "content-length": "682", "date": "Thu, 08 May 2025 22:11:38 GMT"} })
96302025-05-08T22:11:39.088ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
96312025-05-08T22:11:39.088ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
96322025-05-08T22:11:39.092ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
96332025-05-08T22:11:39.092ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
96342025-05-08T22:11:39.096ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
96352025-05-08T22:11:39.096ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
96362025-05-08T22:11:39.098ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
96372025-05-08T22:11:39.098ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
96382025-05-08T22:11:39.105ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 819.739128ms iteration = 27 status = {}
96392025-05-08T22:11:41.940ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 263 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 25c225dc-5faa-4bd0-908b-2146862ca707 response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
96402025-05-08T22:11:48.272ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 80 reason = Timeout
96412025-05-08T22:11:48.272ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
96422025-05-08T22:11:48.272ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
96432025-05-08T22:11:48.274ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96442025-05-08T22:11:48.276ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
96452025-05-08T22:11:48.276ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
96462025-05-08T22:11:48.277ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
96472025-05-08T22:11:48.279ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
96482025-05-08T22:11:48.279ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
96492025-05-08T22:11:48.281ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96502025-05-08T22:11:48.284ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
96512025-05-08T22:11:48.284ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
96522025-05-08T22:11:48.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
96532025-05-08T22:11:48.284ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 80 reason = Timeout
96542025-05-08T22:11:48.284ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
96552025-05-08T22:11:48.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
96562025-05-08T22:11:48.293ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
96572025-05-08T22:11:48.293ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.777132ms iteration = 80 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
96582025-05-08T22:11:48.316ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
96592025-05-08T22:11:48.316ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
96602025-05-08T22:11:48.354ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 69.845787ms iteration = 80 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
96612025-05-08T22:11:51.939ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 260 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 92166821-e7c1-4786-8b62-efe32a357396 response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
96622025-05-08T22:11:58.272ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 81 reason = Timeout
96632025-05-08T22:11:58.272ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
96642025-05-08T22:11:58.272ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
96652025-05-08T22:11:58.284ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 81 reason = Timeout
96662025-05-08T22:11:58.284ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
96672025-05-08T22:11:58.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
96682025-05-08T22:11:58.287ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
96692025-05-08T22:11:58.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
96702025-05-08T22:11:58.288ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.521228ms iteration = 81 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
96712025-05-08T22:11:58.304ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96722025-05-08T22:11:58.307ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
96732025-05-08T22:11:58.307ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
96742025-05-08T22:11:58.307ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
96752025-05-08T22:11:58.310ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
96762025-05-08T22:11:58.310ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
96772025-05-08T22:11:58.311ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96782025-05-08T22:11:58.314ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
96792025-05-08T22:11:58.314ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
96802025-05-08T22:11:58.314ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
96812025-05-08T22:11:58.324ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
96822025-05-08T22:11:58.324ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 52.186778ms iteration = 81 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
96832025-05-08T22:12:01.941ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 256 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 98a57b70-bf25-485d-984a-d2e232eb229c response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
96842025-05-08T22:12:08.270ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 28 reason = Timeout
96852025-05-08T22:12:08.271ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
96862025-05-08T22:12:08.271ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 28 reason = Timeout
96872025-05-08T22:12:08.271ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
96882025-05-08T22:12:08.271ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
96892025-05-08T22:12:08.271ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
96902025-05-08T22:12:08.271ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 82 reason = Timeout
96912025-05-08T22:12:08.271ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 28 reason = Timeout
96922025-05-08T22:12:08.271ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
96932025-05-08T22:12:08.272ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
96942025-05-08T22:12:08.273ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1704, tv_nsec: 10456151 } })
96952025-05-08T22:12:08.273ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 28 reason = Timeout
96962025-05-08T22:12:08.273ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
96972025-05-08T22:12:08.273ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1163
96982025-05-08T22:12:08.274ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1704, tv_nsec: 10894855 } })
96992025-05-08T22:12:08.274ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
97002025-05-08T22:12:08.274ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163
97012025-05-08T22:12:08.274ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97022025-05-08T22:12:08.275ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97032025-05-08T22:12:08.276ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.353226ms iteration = 28 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
97042025-05-08T22:12:08.277ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
97052025-05-08T22:12:08.277ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
97062025-05-08T22:12:08.277ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
97072025-05-08T22:12:08.278ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
97082025-05-08T22:12:08.278ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
97092025-05-08T22:12:08.278ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
97102025-05-08T22:12:08.279ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
97112025-05-08T22:12:08.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
97122025-05-08T22:12:08.283ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97132025-05-08T22:12:08.285ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 28 reason = Timeout
97142025-05-08T22:12:08.285ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 82 reason = Timeout
97152025-05-08T22:12:08.285ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
97162025-05-08T22:12:08.285ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 28 reason = Timeout
97172025-05-08T22:12:08.285ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
97182025-05-08T22:12:08.285ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
97192025-05-08T22:12:08.285ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 28 reason = Timeout
97202025-05-08T22:12:08.285ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
97212025-05-08T22:12:08.285ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
97222025-05-08T22:12:08.285ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
97232025-05-08T22:12:08.285ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 28 reason = Timeout
97242025-05-08T22:12:08.285ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
97252025-05-08T22:12:08.285ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 30 reason = Timeout
97262025-05-08T22:12:08.285ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
97272025-05-08T22:12:08.285ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 28 reason = Timeout
97282025-05-08T22:12:08.285ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
97292025-05-08T22:12:08.285ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 28 reason = Timeout
97302025-05-08T22:12:08.285ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
97312025-05-08T22:12:08.285ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 28 reason = Timeout
97322025-05-08T22:12:08.285ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
97332025-05-08T22:12:08.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
97342025-05-08T22:12:08.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
97352025-05-08T22:12:08.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
97362025-05-08T22:12:08.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
97372025-05-08T22:12:08.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
97382025-05-08T22:12:08.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
97392025-05-08T22:12:08.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
97402025-05-08T22:12:08.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
97412025-05-08T22:12:08.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
97422025-05-08T22:12:08.287ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
97432025-05-08T22:12:08.287ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
97442025-05-08T22:12:08.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
97452025-05-08T22:12:08.287ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 28 reason = Timeout
97462025-05-08T22:12:08.287ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 28 reason = Timeout
97472025-05-08T22:12:08.287ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
97482025-05-08T22:12:08.287ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
97492025-05-08T22:12:08.287ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 28 reason = Timeout
97502025-05-08T22:12:08.287ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
97512025-05-08T22:12:08.287ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 28 reason = Timeout
97522025-05-08T22:12:08.287ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
97532025-05-08T22:12:08.287ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 28 reason = Timeout
97542025-05-08T22:12:08.287ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
97552025-05-08T22:12:08.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
97562025-05-08T22:12:08.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
97572025-05-08T22:12:08.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
97582025-05-08T22:12:08.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
97592025-05-08T22:12:08.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
97602025-05-08T22:12:08.289ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97612025-05-08T22:12:08.291ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
97622025-05-08T22:12:08.291ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
97632025-05-08T22:12:08.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
97642025-05-08T22:12:08.292ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97652025-05-08T22:12:08.292ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97662025-05-08T22:12:08.296ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
97672025-05-08T22:12:08.296ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
97682025-05-08T22:12:08.296ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
97692025-05-08T22:12:08.297ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97702025-05-08T22:12:08.297ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
97712025-05-08T22:12:08.297ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
97722025-05-08T22:12:08.297ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
97732025-05-08T22:12:08.300ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
97742025-05-08T22:12:08.300ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
97752025-05-08T22:12:08.300ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
97762025-05-08T22:12:08.302ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
97772025-05-08T22:12:08.303ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
97782025-05-08T22:12:08.306ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97792025-05-08T22:12:08.308ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
97802025-05-08T22:12:08.308ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
97812025-05-08T22:12:08.308ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
97822025-05-08T22:12:08.308ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
97832025-05-08T22:12:08.309ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
97842025-05-08T22:12:08.309ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
97852025-05-08T22:12:08.309ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 22.261919ms iteration = 28 status = {"errors":[],"requests_created_ok":[]}
97862025-05-08T22:12:08.310ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1704, tv_nsec: 46907711 } })
97872025-05-08T22:12:08.310ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
97882025-05-08T22:12:08.310ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
97892025-05-08T22:12:08.321ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
97902025-05-08T22:12:08.321ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 49.661431ms iteration = 82 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
97912025-05-08T22:12:08.345ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
97922025-05-08T22:12:08.345ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
97932025-05-08T22:12:08.346ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
97942025-05-08T22:12:08.346ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
97952025-05-08T22:12:08.347ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "86032625-d34a-41bb-b306-5deb19e70d1c", "content-length": "26", "date": "Thu, 08 May 2025 22:12:08 GMT"} })
97962025-05-08T22:12:08.347ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
97972025-05-08T22:12:08.350ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
97982025-05-08T22:12:08.354ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97992025-05-08T22:12:08.358ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
98002025-05-08T22:12:08.358ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
98012025-05-08T22:12:08.358ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
98022025-05-08T22:12:08.365ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
98032025-05-08T22:12:08.366ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
98042025-05-08T22:12:08.366ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
98052025-05-08T22:12:08.366ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
98062025-05-08T22:12:08.366ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
98072025-05-08T22:12:08.366ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
98082025-05-08T22:12:08.367ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 79.662441ms iteration = 28 status = {"errors":[],"finish_invoked_ok":[]}
98092025-05-08T22:12:08.371ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 85.896549ms iteration = 82 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
98102025-05-08T22:12:08.371ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
98112025-05-08T22:12:08.371ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
98122025-05-08T22:12:08.371ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
98132025-05-08T22:12:08.372ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
98142025-05-08T22:12:08.373ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 85.683316ms iteration = 28 status = {"errors":[],"garbage_collect_requested":[]}
98152025-05-08T22:12:08.374ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
98162025-05-08T22:12:08.374ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
98172025-05-08T22:12:08.374ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
98182025-05-08T22:12:08.374ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
98192025-05-08T22:12:08.374ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
98202025-05-08T22:12:08.374ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
98212025-05-08T22:12:08.375ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
98222025-05-08T22:12:08.375ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
98232025-05-08T22:12:08.375ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "e602ca5c-c9fe-4689-bf3f-f385fc622879", "content-length": "26", "date": "Thu, 08 May 2025 22:12:08 GMT"} })
98242025-05-08T22:12:08.375ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
98252025-05-08T22:12:08.376ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98262025-05-08T22:12:08.377ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98272025-05-08T22:12:08.378ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
98282025-05-08T22:12:08.379ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 2f2f2c32-8d58-4801-a669-9618eac0c3f7, lookup_type: ById(2f2f2c32-8d58-4801-a669-9618eac0c3f7) } result = Ok(())
98292025-05-08T22:12:08.380ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
98302025-05-08T22:12:08.380ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
98312025-05-08T22:12:08.381ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
98322025-05-08T22:12:08.382ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
98332025-05-08T22:12:08.382ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
98342025-05-08T22:12:08.382ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 96.861775ms iteration = 28 status = 0
98352025-05-08T22:12:08.404ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e zone_address = fd00:1122:3344:101::2
98362025-05-08T22:12:08.405ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
98372025-05-08T22:12:08.405ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b8af55f-0366-404b-b043-506571c5e5d6", "content-length": "26", "date": "Thu, 08 May 2025 22:12:08 GMT"} })
98382025-05-08T22:12:08.405ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1174 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
98392025-05-08T22:12:08.407ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
98402025-05-08T22:12:08.410ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98412025-05-08T22:12:08.410ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
98422025-05-08T22:12:08.410ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
98432025-05-08T22:12:08.410ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
98442025-05-08T22:12:08.410ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
98452025-05-08T22:12:08.412ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 124.883565ms iteration = 28 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
98462025-05-08T22:12:08.413ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
98472025-05-08T22:12:08.413ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
98482025-05-08T22:12:08.413ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
98492025-05-08T22:12:08.415ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:362
98502025-05-08T22:12:08.415ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 141.599535ms iteration = 28 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
98512025-05-08T22:12:08.429ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 143.94087ms iteration = 28 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
98522025-05-08T22:12:08.429ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 142.178271ms iteration = 28 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
98532025-05-08T22:12:08.435ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:552 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
98542025-05-08T22:12:08.435ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 149.684381ms iteration = 28 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
98552025-05-08T22:12:08.448ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
98562025-05-08T22:12:08.448ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
98572025-05-08T22:12:08.448ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 163.409818ms iteration = 28 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
98582025-05-08T22:12:08.468ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
98592025-05-08T22:12:08.468ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
98602025-05-08T22:12:08.470ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 200.008539ms iteration = 28 status = {}
98612025-05-08T22:12:08.526ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 240.748305ms iteration = 30 status = {"physical_disks_added":0}
98622025-05-08T22:12:08.530ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
98632025-05-08T22:12:08.531ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
98642025-05-08T22:12:08.533ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 262.935173ms iteration = 28 status = null
98652025-05-08T22:12:08.568ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
98662025-05-08T22:12:08.568ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
98672025-05-08T22:12:08.568ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "501f64e3-9b56-4f8b-8d6b-10b48e20beca", "content-length": "372", "date": "Thu, 08 May 2025 22:12:08 GMT"} })
98682025-05-08T22:12:08.568ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
98692025-05-08T22:12:08.568ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
98702025-05-08T22:12:08.586ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
98712025-05-08T22:12:08.586ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
98722025-05-08T22:12:08.587ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "79898a27-48ad-493c-aaf0-c398dc8397b8", "date": "Thu, 08 May 2025 22:12:08 GMT"} })
98732025-05-08T22:12:08.587ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
98742025-05-08T22:12:08.587ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
98752025-05-08T22:12:08.590ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
98762025-05-08T22:12:08.591ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
98772025-05-08T22:12:08.594ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
98782025-05-08T22:12:08.594ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
98792025-05-08T22:12:08.598ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
98802025-05-08T22:12:08.598ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
98812025-05-08T22:12:08.598ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "38df3411-9beb-4a08-874d-311998057533", "date": "Thu, 08 May 2025 22:12:08 GMT"} })
98822025-05-08T22:12:08.600ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 314.744586ms iteration = 28 status = {}
98832025-05-08T22:12:08.610ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
98842025-05-08T22:12:08.610ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
98852025-05-08T22:12:08.612ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98862025-05-08T22:12:08.614ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
98872025-05-08T22:12:08.614ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
98882025-05-08T22:12:08.614ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
98892025-05-08T22:12:08.615ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98902025-05-08T22:12:08.617ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
98912025-05-08T22:12:08.617ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
98922025-05-08T22:12:08.617ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
98932025-05-08T22:12:08.617ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
98942025-05-08T22:12:08.617ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
98952025-05-08T22:12:08.619ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: cd0b930c-534b-46ad-ac3b-2eaa94b0bf2c, rack_id: 64f3d6f4-89b8-48aa-a622-b684e261667e, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(65f83a36-18fb-49b7-9914-87c8517d200e) } rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
98962025-05-08T22:12:08.619ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
98972025-05-08T22:12:08.619ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
98982025-05-08T22:12:08.641ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1833 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
98992025-05-08T22:12:08.641ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1860 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 65f83a36-18fb-49b7-9914-87c8517d200e
99002025-05-08T22:12:08.641ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
99012025-05-08T22:12:08.641ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
99022025-05-08T22:12:08.641ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
99032025-05-08T22:12:08.641ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
99042025-05-08T22:12:08.641ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
99052025-05-08T22:12:08.641ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2007 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
99062025-05-08T22:12:08.641ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
99072025-05-08T22:12:08.641ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2043 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
99082025-05-08T22:12:08.641ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
99092025-05-08T22:12:08.641ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
99102025-05-08T22:12:08.642ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
99112025-05-08T22:12:08.644ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
99122025-05-08T22:12:08.644ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
99132025-05-08T22:12:08.721ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
99142025-05-08T22:12:08.748ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "81aa3e8e-e774-4570-a292-67e53f1ebcec", "date": "Thu, 08 May 2025 22:12:08 GMT"} })
99152025-05-08T22:12:08.748ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch = Switch0
99162025-05-08T22:12:08.748ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
99172025-05-08T22:12:08.748ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "f412c27a-f06d-4260-82f7-b81faff294aa", "content-length": "691", "date": "Thu, 08 May 2025 22:12:08 GMT"} })
99182025-05-08T22:12:08.748ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
99192025-05-08T22:12:08.748ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
99202025-05-08T22:12:08.748ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
99212025-05-08T22:12:08.748ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "f262f475-bea5-4633-9639-3ac7e36706fd", "content-length": "682", "date": "Thu, 08 May 2025 22:12:08 GMT"} })
99222025-05-08T22:12:08.748ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
99232025-05-08T22:12:08.749ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
99242025-05-08T22:12:08.749ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 464.074293ms iteration = 28 status = {}
99252025-05-08T22:12:08.752ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
99262025-05-08T22:12:08.753ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
99272025-05-08T22:12:08.756ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
99282025-05-08T22:12:08.756ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
99292025-05-08T22:12:08.758ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
99302025-05-08T22:12:08.759ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
99312025-05-08T22:12:08.765ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 480.066815ms iteration = 28 status = {}
99322025-05-08T22:12:09.340ZDEBGnexus (producer-registration-task): registering as oximeter producer with Nexus registration_count = 5
99332025-05-08T22:12:09.340ZDEBGnexus (producer-registration-task): will register with Nexus at [fd00:1122:3344:101::b]:12221
99342025-05-08T22:12:09.370ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::b]:12221/metrics/producers
99352025-05-08T22:12:09.370ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::b]:40941
99362025-05-08T22:12:09.370ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:40941 req_id = 4072a215-1b4b-4ece-8328-ff33d81c2718 roles = RoleSet { roles: {} } uri = /metrics/producers
99372025-05-08T22:12:09.370ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:40941 req_id = 4072a215-1b4b-4ece-8328-ff33d81c2718 resource = Database result = Ok(()) uri = /metrics/producers
99382025-05-08T22:12:09.414ZINFOnexus: assigned collector to new producer collector_id = 22ae79e2-3731-44c9-8346-78c5ddbfe577 file = nexus/src/app/oximeter.rs:76 producer_id = 85a39cf9-9419-458f-af2e-c8b40d8f5f79
99392025-05-08T22:12:09.414ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 43638 local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:40941 req_id = 4072a215-1b4b-4ece-8328-ff33d81c2718 response_code = 201 uri = /metrics/producers
99402025-05-08T22:12:09.414ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "4072a215-1b4b-4ece-8328-ff33d81c2718", "content-length": "41", "date": "Thu, 08 May 2025 22:12:09 GMT"} })
99412025-05-08T22:12:09.416ZDEBGnexus (producer-registration-task): successfully registered with Nexus, pausing until time to renew lease lease_duration = 600s wait_period = 150s
99422025-05-08T22:12:11.940ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 257 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 15b24ba8-b46b-4551-b055-accd2eb88dba response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
99432025-05-08T22:12:18.272ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 83 reason = Timeout
99442025-05-08T22:12:18.272ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
99452025-05-08T22:12:18.273ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
99462025-05-08T22:12:18.275ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99472025-05-08T22:12:18.277ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
99482025-05-08T22:12:18.277ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
99492025-05-08T22:12:18.277ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
99502025-05-08T22:12:18.280ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
99512025-05-08T22:12:18.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
99522025-05-08T22:12:18.281ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99532025-05-08T22:12:18.284ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
99542025-05-08T22:12:18.284ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
99552025-05-08T22:12:18.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
99562025-05-08T22:12:18.285ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 83 reason = Timeout
99572025-05-08T22:12:18.285ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
99582025-05-08T22:12:18.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
99592025-05-08T22:12:18.287ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
99602025-05-08T22:12:18.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
99612025-05-08T22:12:18.292ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
99622025-05-08T22:12:18.292ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.450969ms iteration = 83 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
99632025-05-08T22:12:18.316ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 31.489927ms iteration = 83 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
99642025-05-08T22:12:21.940ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 255 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = df9f31ab-62b1-43fb-a841-2133fde7595a response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
99652025-05-08T22:12:28.272ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 84 reason = Timeout
99662025-05-08T22:12:28.272ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
99672025-05-08T22:12:28.272ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
99682025-05-08T22:12:28.285ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 84 reason = Timeout
99692025-05-08T22:12:28.285ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
99702025-05-08T22:12:28.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
99712025-05-08T22:12:28.316ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99722025-05-08T22:12:28.318ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
99732025-05-08T22:12:28.318ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
99742025-05-08T22:12:28.319ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
99752025-05-08T22:12:28.319ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
99762025-05-08T22:12:28.319ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
99772025-05-08T22:12:28.320ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 34.885303ms iteration = 84 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
99782025-05-08T22:12:28.321ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
99792025-05-08T22:12:28.321ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
99802025-05-08T22:12:28.323ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99812025-05-08T22:12:28.326ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
99822025-05-08T22:12:28.326ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
99832025-05-08T22:12:28.326ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
99842025-05-08T22:12:28.336ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
99852025-05-08T22:12:28.336ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 64.365409ms iteration = 84 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
99862025-05-08T22:12:31.940ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 435 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 76757b27-891e-4cd4-988c-50383c9234a4 response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
99872025-05-08T22:12:36.933ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::d]:49336
99882025-05-08T22:12:36.933ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49336 req_id = 135a01e3-c2d4-49f6-853c-af2927205845 roles = RoleSet { roles: {} } uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?sort_by=id_ascending
99892025-05-08T22:12:36.934ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49336 req_id = 135a01e3-c2d4-49f6-853c-af2927205845 resource = Database result = Ok(()) uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?sort_by=id_ascending
99902025-05-08T22:12:36.937ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3467 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49336 req_id = 135a01e3-c2d4-49f6-853c-af2927205845 response_code = 200 uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?sort_by=id_ascending
99912025-05-08T22:12:36.937ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49336 req_id = c4bc130a-cc69-4efa-8d06-382ad343fe34 roles = RoleSet { roles: {} } uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZGQ4MjYzNC0yODQ0LTRmOTctOTc0Ni0wNzA5OTA0NGMxOWQifX0%3D
99922025-05-08T22:12:36.937ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49336 req_id = c4bc130a-cc69-4efa-8d06-382ad343fe34 resource = Database result = Ok(()) uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZGQ4MjYzNC0yODQ0LTRmOTctOTc0Ni0wNzA5OTA0NGMxOWQifX0%3D
99932025-05-08T22:12:36.940ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2684 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49336 req_id = c4bc130a-cc69-4efa-8d06-382ad343fe34 response_code = 200 uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZGQ4MjYzNC0yODQ0LTRmOTctOTc0Ni0wNzA5OTA0NGMxOWQifX0%3D
99942025-05-08T22:12:38.270ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 29 reason = Timeout
99952025-05-08T22:12:38.270ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
99962025-05-08T22:12:38.270ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 17 reason = Timeout
99972025-05-08T22:12:38.270ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 15 reason = Timeout
99982025-05-08T22:12:38.270ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 15 reason = Timeout
99992025-05-08T22:12:38.270ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
100002025-05-08T22:12:38.270ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-05-08 22:02:38.270711960 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
100012025-05-08T22:12:38.270ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
100022025-05-08T22:12:38.270ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 15 reason = Timeout
100032025-05-08T22:12:38.270ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 17 reason = Timeout
100042025-05-08T22:12:38.270ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 15 reason = Timeout
100052025-05-08T22:12:38.270ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
100062025-05-08T22:12:38.270ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
100072025-05-08T22:12:38.270ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 15 reason = Timeout
100082025-05-08T22:12:38.270ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 29 reason = Timeout
100092025-05-08T22:12:38.270ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
100102025-05-08T22:12:38.270ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 15 reason = Timeout
100112025-05-08T22:12:38.270ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
100122025-05-08T22:12:38.270ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
100132025-05-08T22:12:38.271ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
100142025-05-08T22:12:38.271ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
100152025-05-08T22:12:38.271ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
100162025-05-08T22:12:38.271ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
100172025-05-08T22:12:38.271ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
100182025-05-08T22:12:38.271ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("228ed1c1-3cc1-4055-aae9-5595ad856b86.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("8b245480-f91d-406b-bb08-850e1c3e3fc5.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c8f20f67-ea63-4dbc-b0f1-c30876daff82.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c8f20f67-ea63-4dbc-b0f1-c30876daff82.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 1734, tv_nsec: 7987354 } })
100192025-05-08T22:12:38.271ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("0ccabf41-81cd-45e7-bb26-335edaa2ee47.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("38a08dd6-6a75-4558-b4f8-df853f4d03d9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("38a08dd6-6a75-4558-b4f8-df853f4d03d9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::9))) }], valid_until: Instant { tv_sec: 1734, tv_nsec: 8018524 } })
100202025-05-08T22:12:38.271ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
100212025-05-08T22:12:38.271ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } dns_group = external
100222025-05-08T22:12:38.271ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.125332ms iteration = 15 status = {"addresses":["[fd00:1122:3344:101::8]:5353","[fd00:1122:3344:101::9]:5353"]}
100232025-05-08T22:12:38.272ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
100242025-05-08T22:12:38.272ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.216233ms iteration = 15 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:1::1]:5353"]}
100252025-05-08T22:12:38.272ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
100262025-05-08T22:12:38.272ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
100272025-05-08T22:12:38.272ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 85 reason = Timeout
100282025-05-08T22:12:38.272ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 16 reason = Timeout
100292025-05-08T22:12:38.272ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
100302025-05-08T22:12:38.272ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:85 reason = blueprint disabled target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2
100312025-05-08T22:12:38.272ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 29 reason = Timeout
100322025-05-08T22:12:38.272ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 10.22\u{b5}s iteration = 16 status = {"enabled":false,"target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2"}
100332025-05-08T22:12:38.273ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
100342025-05-08T22:12:38.273ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1734, tv_nsec: 10378070 } })
100352025-05-08T22:12:38.273ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
100362025-05-08T22:12:38.273ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1163
100372025-05-08T22:12:38.274ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
100382025-05-08T22:12:38.274ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 29 reason = Timeout
100392025-05-08T22:12:38.274ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
100402025-05-08T22:12:38.274ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
100412025-05-08T22:12:38.274ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
100422025-05-08T22:12:38.274ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1734, tv_nsec: 11307720 } })
100432025-05-08T22:12:38.274ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163
100442025-05-08T22:12:38.275ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-05-08 22:02:38.270711960 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
100452025-05-08T22:12:38.275ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 4.735301ms iteration = 15 status = {"expiration":"2025-05-08T22:02:38.270711960Z","pruned":[]}
100462025-05-08T22:12:38.276ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
100472025-05-08T22:12:38.276ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
100482025-05-08T22:12:38.276ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
100492025-05-08T22:12:38.278ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
100502025-05-08T22:12:38.279ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
100512025-05-08T22:12:38.279ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
100522025-05-08T22:12:38.280ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 7.760783ms iteration = 29 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
100532025-05-08T22:12:38.281ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
100542025-05-08T22:12:38.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
100552025-05-08T22:12:38.282ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
100562025-05-08T22:12:38.283ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
100572025-05-08T22:12:38.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
100582025-05-08T22:12:38.283ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
100592025-05-08T22:12:38.283ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
100602025-05-08T22:12:38.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
100612025-05-08T22:12:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
100622025-05-08T22:12:38.285ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 31 reason = Timeout
100632025-05-08T22:12:38.285ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 29 reason = Timeout
100642025-05-08T22:12:38.285ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
100652025-05-08T22:12:38.285ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
100662025-05-08T22:12:38.285ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 29 reason = Timeout
100672025-05-08T22:12:38.285ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 85 reason = Timeout
100682025-05-08T22:12:38.285ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
100692025-05-08T22:12:38.285ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
100702025-05-08T22:12:38.285ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 29 reason = Timeout
100712025-05-08T22:12:38.285ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
100722025-05-08T22:12:38.285ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 15 reason = Timeout
100732025-05-08T22:12:38.285ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:53
100742025-05-08T22:12:38.285ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
100752025-05-08T22:12:38.285ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 29 reason = Timeout
100762025-05-08T22:12:38.285ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
100772025-05-08T22:12:38.285ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
100782025-05-08T22:12:38.285ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 29 reason = Timeout
100792025-05-08T22:12:38.285ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 15 reason = Timeout
100802025-05-08T22:12:38.285ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
100812025-05-08T22:12:38.285ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
100822025-05-08T22:12:38.285ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 29 reason = Timeout
100832025-05-08T22:12:38.285ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
100842025-05-08T22:12:38.285ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 29 reason = Timeout
100852025-05-08T22:12:38.285ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
100862025-05-08T22:12:38.285ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
100872025-05-08T22:12:38.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
100882025-05-08T22:12:38.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
100892025-05-08T22:12:38.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
100902025-05-08T22:12:38.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
100912025-05-08T22:12:38.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
100922025-05-08T22:12:38.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
100932025-05-08T22:12:38.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
100942025-05-08T22:12:38.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
100952025-05-08T22:12:38.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
100962025-05-08T22:12:38.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
100972025-05-08T22:12:38.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
100982025-05-08T22:12:38.287ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
100992025-05-08T22:12:38.287ZDEBGnexus (BackgroundTasks): activating background_task = webhook_deliverator iteration = 15 reason = Timeout
101002025-05-08T22:12:38.287ZDEBGnexus (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} }
101012025-05-08T22:12:38.287ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 15 reason = Timeout
101022025-05-08T22:12:38.287ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
101032025-05-08T22:12:38.287ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 29 reason = Timeout
101042025-05-08T22:12:38.287ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
101052025-05-08T22:12:38.287ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 29 reason = Timeout
101062025-05-08T22:12:38.287ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
101072025-05-08T22:12:38.287ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 29 reason = Timeout
101082025-05-08T22:12:38.287ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
101092025-05-08T22:12:38.287ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 29 reason = Timeout
101102025-05-08T22:12:38.287ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
101112025-05-08T22:12:38.287ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 15 reason = Timeout
101122025-05-08T22:12:38.287ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
101132025-05-08T22:12:38.287ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 29 reason = Timeout
101142025-05-08T22:12:38.287ZDEBGnexus (BackgroundTasks): activating background_task = webhook_dispatcher iteration = 15 reason = Timeout
101152025-05-08T22:12:38.287ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
101162025-05-08T22:12:38.287ZDEBGnexus (BackgroundTasks): roles background_task = webhook_dispatcher roles = RoleSet { roles: {} }
101172025-05-08T22:12:38.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
101182025-05-08T22:12:38.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
101192025-05-08T22:12:38.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database result = Ok(())
101202025-05-08T22:12:38.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
101212025-05-08T22:12:38.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
101222025-05-08T22:12:38.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
101232025-05-08T22:12:38.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
101242025-05-08T22:12:38.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
101252025-05-08T22:12:38.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database result = Ok(())
101262025-05-08T22:12:38.290ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
101272025-05-08T22:12:38.290ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
101282025-05-08T22:12:38.291ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
101292025-05-08T22:12:38.296ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101302025-05-08T22:12:38.296ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101312025-05-08T22:12:38.297ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
101322025-05-08T22:12:38.297ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101332025-05-08T22:12:38.298ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
101342025-05-08T22:12:38.300ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
101352025-05-08T22:12:38.301ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
101362025-05-08T22:12:38.301ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
101372025-05-08T22:12:38.301ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
101382025-05-08T22:12:38.301ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
101392025-05-08T22:12:38.301ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
101402025-05-08T22:12:38.301ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
101412025-05-08T22:12:38.301ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
101422025-05-08T22:12:38.302ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
101432025-05-08T22:12:38.304ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
101442025-05-08T22:12:38.305ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
101452025-05-08T22:12:38.305ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 18.198615ms iteration = 15 status = {"errors":[],"found_port_ok":[]}
101462025-05-08T22:12:38.305ZDEBGnexus (BackgroundTasks): activation complete background_task = webhook_deliverator elapsed = 18.419017ms iteration = 15 status = {"by_rx":{},"error":null}
101472025-05-08T22:12:38.311ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1734, tv_nsec: 48240705 } })
101482025-05-08T22:12:38.317ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
101492025-05-08T22:12:38.341ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101502025-05-08T22:12:38.343ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
101512025-05-08T22:12:38.343ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
101522025-05-08T22:12:38.348ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
101532025-05-08T22:12:38.350ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
101542025-05-08T22:12:38.350ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
101552025-05-08T22:12:38.350ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
101562025-05-08T22:12:38.350ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
101572025-05-08T22:12:38.350ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
101582025-05-08T22:12:38.352ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 67.245629ms iteration = 85 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
101592025-05-08T22:12:38.353ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
101602025-05-08T22:12:38.354ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
101612025-05-08T22:12:38.354ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101622025-05-08T22:12:38.357ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
101632025-05-08T22:12:38.357ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
101642025-05-08T22:12:38.357ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
101652025-05-08T22:12:38.357ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
101662025-05-08T22:12:38.357ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
101672025-05-08T22:12:38.357ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
101682025-05-08T22:12:38.357ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
101692025-05-08T22:12:38.357ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101702025-05-08T22:12:38.357ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
101712025-05-08T22:12:38.359ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
101722025-05-08T22:12:38.359ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
101732025-05-08T22:12:38.359ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
101742025-05-08T22:12:38.359ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
101752025-05-08T22:12:38.360ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101762025-05-08T22:12:38.362ZDEBGnexus (BackgroundTasks): roles background_task = webhook_dispatcher roles = RoleSet { roles: {} }
101772025-05-08T22:12:38.362ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
101782025-05-08T22:12:38.362ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
101792025-05-08T22:12:38.362ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
101802025-05-08T22:12:38.362ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database result = Ok(())
101812025-05-08T22:12:38.366ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
101822025-05-08T22:12:38.366ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
101832025-05-08T22:12:38.366ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
101842025-05-08T22:12:38.369ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101852025-05-08T22:12:38.370ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101862025-05-08T22:12:38.370ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101872025-05-08T22:12:38.374ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
101882025-05-08T22:12:38.374ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
101892025-05-08T22:12:38.374ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
101902025-05-08T22:12:38.375ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
101912025-05-08T22:12:38.375ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
101922025-05-08T22:12:38.375ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
101932025-05-08T22:12:38.375ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101942025-05-08T22:12:38.378ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
101952025-05-08T22:12:38.378ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
101962025-05-08T22:12:38.378ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101972025-05-08T22:12:38.378ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
101982025-05-08T22:12:38.378ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 93.618041ms iteration = 29 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
101992025-05-08T22:12:38.379ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
102002025-05-08T22:12:38.379ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
102012025-05-08T22:12:38.379ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
102022025-05-08T22:12:38.381ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
102032025-05-08T22:12:38.381ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
102042025-05-08T22:12:38.381ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
102052025-05-08T22:12:38.381ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
102062025-05-08T22:12:38.381ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
102072025-05-08T22:12:38.382ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
102082025-05-08T22:12:38.382ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
102092025-05-08T22:12:38.382ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
102102025-05-08T22:12:38.382ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
102112025-05-08T22:12:38.382ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 7586992a-6fc5-4073-8962-728778f16aa5 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
102122025-05-08T22:12:38.382ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
102132025-05-08T22:12:38.382ZDEBGnexus (BackgroundTasks): activation complete background_task = webhook_dispatcher elapsed = 95.056156ms iteration = 15 status = {"dispatched":[],"errors":[],"glob_version":"140.0.0","globs_reprocessed":{},"no_receivers":[]}
102142025-05-08T22:12:38.382ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
102152025-05-08T22:12:38.382ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
102162025-05-08T22:12:38.386ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 98.676435ms iteration = 29 status = {"errors":[],"garbage_collect_requested":[]}
102172025-05-08T22:12:38.386ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
102182025-05-08T22:12:38.386ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
102192025-05-08T22:12:38.386ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 98.762216ms iteration = 15 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
102202025-05-08T22:12:38.387ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 100.333283ms iteration = 29 status = {"errors":[],"finish_invoked_ok":[]}
102212025-05-08T22:12:38.391ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
102222025-05-08T22:12:38.391ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = d8128696-db5a-4b88-914a-69b28494e50e dns_zone_name = oxide.test found_so_far = 0 version = 2
102232025-05-08T22:12:38.391ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
102242025-05-08T22:12:38.392ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
102252025-05-08T22:12:38.394ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102262025-05-08T22:12:38.394ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
102272025-05-08T22:12:38.394ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
102282025-05-08T22:12:38.395ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 107.709901ms iteration = 29 status = {"errors":[],"requests_created_ok":[]}
102292025-05-08T22:12:38.398ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:552 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
102302025-05-08T22:12:38.398ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 112.679445ms iteration = 29 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
102312025-05-08T22:12:38.399ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
102322025-05-08T22:12:38.399ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
102332025-05-08T22:12:38.399ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
102342025-05-08T22:12:38.403ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 116.246973ms iteration = 29 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
102352025-05-08T22:12:38.409ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
102362025-05-08T22:12:38.409ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:137 found = 0 not_ready_to_be_deleted = 0
102372025-05-08T22:12:38.409ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 124.499322ms iteration = 15 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
102382025-05-08T22:12:38.412ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
102392025-05-08T22:12:38.413ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
102402025-05-08T22:12:38.416ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
102412025-05-08T22:12:38.416ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 143.124021ms iteration = 85 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
102422025-05-08T22:12:38.423ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
102432025-05-08T22:12:38.424ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
102442025-05-08T22:12:38.424ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:362
102452025-05-08T22:12:38.424ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 150.325557ms iteration = 29 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
102462025-05-08T22:12:38.427ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
102472025-05-08T22:12:38.427ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
102482025-05-08T22:12:38.430ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
102492025-05-08T22:12:38.430ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102502025-05-08T22:12:38.431ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
102512025-05-08T22:12:38.434ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
102522025-05-08T22:12:38.434ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
102532025-05-08T22:12:38.434ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
102542025-05-08T22:12:38.434ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
102552025-05-08T22:12:38.435ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102562025-05-08T22:12:38.439ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
102572025-05-08T22:12:38.439ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
102582025-05-08T22:12:38.439ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
102592025-05-08T22:12:38.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
102602025-05-08T22:12:38.440ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 2f2f2c32-8d58-4801-a669-9618eac0c3f7, lookup_type: ById(2f2f2c32-8d58-4801-a669-9618eac0c3f7) } result = Ok(())
102612025-05-08T22:12:38.440ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
102622025-05-08T22:12:38.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
102632025-05-08T22:12:38.441ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
102642025-05-08T22:12:38.441ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
102652025-05-08T22:12:38.441ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f1290da-0500-49f7-9ea2-f434eed8668a", "content-length": "26", "date": "Thu, 08 May 2025 22:12:38 GMT"} })
102662025-05-08T22:12:38.441ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
102672025-05-08T22:12:38.442ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 157.250742ms iteration = 29 status = 0
102682025-05-08T22:12:38.443ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
102692025-05-08T22:12:38.443ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
102702025-05-08T22:12:38.443ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "6fb2fe0d-bdc7-49f8-8f90-c48f24d2f8a9", "content-length": "26", "date": "Thu, 08 May 2025 22:12:38 GMT"} })
102712025-05-08T22:12:38.443ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
102722025-05-08T22:12:38.443ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
102732025-05-08T22:12:38.445ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
102742025-05-08T22:12:38.445ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
102752025-05-08T22:12:38.450ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102762025-05-08T22:12:38.454ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102772025-05-08T22:12:38.455ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
102782025-05-08T22:12:38.455ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
102792025-05-08T22:12:38.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
102802025-05-08T22:12:38.457ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
102812025-05-08T22:12:38.457ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
102822025-05-08T22:12:38.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
102832025-05-08T22:12:38.459ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 7586992a-6fc5-4073-8962-728778f16aa5 dns_zone_name = control-plane.oxide.internal found_so_far = 55 version = 1
102842025-05-08T22:12:38.459ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
102852025-05-08T22:12:38.459ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-05-08 21:59:21.182334 UTC dns_group = internal generation = 1
102862025-05-08T22:12:38.459ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 188.982221ms iteration = 15 status = {"generation":1}
102872025-05-08T22:12:38.460ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = d8128696-db5a-4b88-914a-69b28494e50e dns_zone_name = oxide.test found_so_far = 1 version = 2
102882025-05-08T22:12:38.460ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
102892025-05-08T22:12:38.460ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-05-08 21:59:21.759368 UTC dns_group = external generation = 2
102902025-05-08T22:12:38.460ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 189.636558ms iteration = 15 status = {"generation":2}
102912025-05-08T22:12:38.465ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 177.928953ms iteration = 29 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
102922025-05-08T22:12:38.465ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
102932025-05-08T22:12:38.465ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
102942025-05-08T22:12:38.466ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e zone_address = fd00:1122:3344:101::2
102952025-05-08T22:12:38.466ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
102962025-05-08T22:12:38.466ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "d232c9d1-8b2a-4eba-bb2b-4af80ce3159f", "content-length": "26", "date": "Thu, 08 May 2025 22:12:38 GMT"} })
102972025-05-08T22:12:38.466ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1174 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
102982025-05-08T22:12:38.468ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
102992025-05-08T22:12:38.471ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
103002025-05-08T22:12:38.471ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
103012025-05-08T22:12:38.485ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
103022025-05-08T22:12:38.485ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
103032025-05-08T22:12:38.485ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 200.229032ms iteration = 29 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
103042025-05-08T22:12:38.493ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-05-08T21:58:35.560911Z, time_modified: 2025-05-08T21:58:35.560911Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, db_silo: Silo { identity: SiloIdentity { id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-05-08T21:59:21.709488Z, time_modified: 2025-05-08T21:59:21.709488Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, db_silo: Silo { identity: SiloIdentity { id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-05-08T21:59:21.709488Z, time_modified: 2025-05-08T21:59:21.709488Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47" }] }) }
103052025-05-08T22:12:38.493ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 222.723562ms iteration = 15 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"7137ee58-6aa7-4199-a248-feadf3b8bafb","tls_certs":["37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47"]}},"default_endpoint":{"silo_id":"7137ee58-6aa7-4199-a248-feadf3b8bafb","tls_certs":["37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
103062025-05-08T22:12:38.535ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 250.013634ms iteration = 31 status = {"physical_disks_added":0}
103072025-05-08T22:12:38.607ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 336.736622ms iteration = 29 status = {}
103082025-05-08T22:12:38.620ZINFOnexus (BackgroundTasks): instance reincarnation completed successfully background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:113 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0
103092025-05-08T22:12:38.620ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 335.335256ms iteration = 15 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
103102025-05-08T22:12:38.633ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
103112025-05-08T22:12:38.634ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
103122025-05-08T22:12:38.635ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103132025-05-08T22:12:38.639ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
103142025-05-08T22:12:38.639ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
103152025-05-08T22:12:38.639ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
103162025-05-08T22:12:38.642ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103172025-05-08T22:12:38.644ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
103182025-05-08T22:12:38.644ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
103192025-05-08T22:12:38.644ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
103202025-05-08T22:12:38.683ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
103212025-05-08T22:12:38.683ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
103222025-05-08T22:12:38.684ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "53c48b6a-c1e4-43cf-ba5e-39d040d782ee", "content-length": "372", "date": "Thu, 08 May 2025 22:12:38 GMT"} })
103232025-05-08T22:12:38.684ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
103242025-05-08T22:12:38.684ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
103252025-05-08T22:12:38.722ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
103262025-05-08T22:12:38.722ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
103272025-05-08T22:12:38.725ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 454.230788ms iteration = 29 status = null
103282025-05-08T22:12:38.737ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
103292025-05-08T22:12:38.738ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
103302025-05-08T22:12:38.738ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "f6852a46-e293-4cf4-bd9e-bc60e69c28e0", "date": "Thu, 08 May 2025 22:12:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
103312025-05-08T22:12:38.738ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
103322025-05-08T22:12:38.740ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "1ce62a65-98cb-4d89-937d-67e3bfa4a544", "date": "Thu, 08 May 2025 22:12:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
103332025-05-08T22:12:38.740ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
103342025-05-08T22:12:38.743ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
103352025-05-08T22:12:38.743ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 472.874687ms iteration = 17 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
103362025-05-08T22:12:38.749ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
103372025-05-08T22:12:38.749ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
103382025-05-08T22:12:38.749ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "de77de52-9c20-4e3c-a20d-328636122dd5", "date": "Thu, 08 May 2025 22:12:38 GMT"} })
103392025-05-08T22:12:38.749ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
103402025-05-08T22:12:38.749ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
103412025-05-08T22:12:38.779ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
103422025-05-08T22:12:38.782ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "9505a77d-d54b-4cc2-97c6-c91f4c003c78", "content-length": "691", "date": "Thu, 08 May 2025 22:12:38 GMT"} })
103432025-05-08T22:12:38.782ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
103442025-05-08T22:12:38.782ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
103452025-05-08T22:12:38.782ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
103462025-05-08T22:12:38.782ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
103472025-05-08T22:12:38.783ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
103482025-05-08T22:12:38.783ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 498.543852ms iteration = 29 status = {}
103492025-05-08T22:12:38.786ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
103502025-05-08T22:12:38.786ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
103512025-05-08T22:12:38.790ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
103522025-05-08T22:12:38.790ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
103532025-05-08T22:12:38.790ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "4e1187c1-dec3-4556-9b52-b0b0b75f3e98", "date": "Thu, 08 May 2025 22:12:38 GMT"} })
103542025-05-08T22:12:38.792ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 506.795961ms iteration = 29 status = {}
103552025-05-08T22:12:38.887ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
103562025-05-08T22:12:38.887ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "acec1257-6039-4dc8-8761-e0d467ae9169", "date": "Thu, 08 May 2025 22:12:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
103572025-05-08T22:12:38.887ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
103582025-05-08T22:12:38.889ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "a9d138b1-dd00-48d4-a0af-e33652c5f176", "date": "Thu, 08 May 2025 22:12:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
103592025-05-08T22:12:38.889ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
103602025-05-08T22:12:38.891ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "e277beec-e267-4199-82b0-5f2560f3c753", "date": "Thu, 08 May 2025 22:12:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
103612025-05-08T22:12:38.891ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
103622025-05-08T22:12:38.894ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
103632025-05-08T22:12:38.894ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 623.448428ms iteration = 17 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
103642025-05-08T22:12:38.920ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
103652025-05-08T22:12:38.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
103662025-05-08T22:12:38.923ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: cd0b930c-534b-46ad-ac3b-2eaa94b0bf2c, rack_id: 64f3d6f4-89b8-48aa-a622-b684e261667e, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(65f83a36-18fb-49b7-9914-87c8517d200e) } rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
103672025-05-08T22:12:38.923ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
103682025-05-08T22:12:38.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
103692025-05-08T22:12:38.932ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1833 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
103702025-05-08T22:12:38.932ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1860 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 65f83a36-18fb-49b7-9914-87c8517d200e
103712025-05-08T22:12:38.932ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
103722025-05-08T22:12:38.932ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
103732025-05-08T22:12:38.932ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
103742025-05-08T22:12:38.932ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
103752025-05-08T22:12:38.932ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
103762025-05-08T22:12:38.932ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2007 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
103772025-05-08T22:12:38.932ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
103782025-05-08T22:12:38.932ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2043 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
103792025-05-08T22:12:38.932ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
103802025-05-08T22:12:38.932ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
103812025-05-08T22:12:38.933ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
103822025-05-08T22:12:38.934ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
103832025-05-08T22:12:38.934ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
103842025-05-08T22:12:39.034ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "03a3a0d1-a2ac-42d4-acf4-e1802b6f6105", "date": "Thu, 08 May 2025 22:12:38 GMT"} })
103852025-05-08T22:12:39.034ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch = Switch0
103862025-05-08T22:12:39.034ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
103872025-05-08T22:12:39.035ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e63f3e0-f27d-44ff-a530-ca48ae7974ca", "content-length": "682", "date": "Thu, 08 May 2025 22:12:39 GMT"} })
103882025-05-08T22:12:39.035ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
103892025-05-08T22:12:39.035ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
103902025-05-08T22:12:39.038ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
103912025-05-08T22:12:39.039ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
103922025-05-08T22:12:39.042ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
103932025-05-08T22:12:39.042ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
103942025-05-08T22:12:39.043ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
103952025-05-08T22:12:39.044ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
103962025-05-08T22:12:39.050ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 765.090703ms iteration = 29 status = {}
103972025-05-08T22:12:41.940ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 309 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 80057ef8-63e6-4a74-b9cd-8bd51fa38f04 response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
103982025-05-08T22:12:48.273ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 86 reason = Timeout
103992025-05-08T22:12:48.273ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
104002025-05-08T22:12:48.273ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
104012025-05-08T22:12:48.275ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
104022025-05-08T22:12:48.278ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
104032025-05-08T22:12:48.278ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
104042025-05-08T22:12:48.278ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
104052025-05-08T22:12:48.280ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
104062025-05-08T22:12:48.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
104072025-05-08T22:12:48.282ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
104082025-05-08T22:12:48.284ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
104092025-05-08T22:12:48.284ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
104102025-05-08T22:12:48.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
104112025-05-08T22:12:48.285ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 86 reason = Timeout
104122025-05-08T22:12:48.285ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
104132025-05-08T22:12:48.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
104142025-05-08T22:12:48.287ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
104152025-05-08T22:12:48.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
104162025-05-08T22:12:48.288ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.419676ms iteration = 86 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
104172025-05-08T22:12:48.294ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
104182025-05-08T22:12:48.294ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.679272ms iteration = 86 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
104192025-05-08T22:12:51.933ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49336 req_id = 68d2eb97-4f3c-40af-9319-18e21bab70b4 roles = RoleSet { roles: {} } uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?sort_by=id_ascending
104202025-05-08T22:12:51.933ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49336 req_id = 68d2eb97-4f3c-40af-9319-18e21bab70b4 resource = Database result = Ok(()) uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?sort_by=id_ascending
104212025-05-08T22:12:51.936ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3285 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49336 req_id = 68d2eb97-4f3c-40af-9319-18e21bab70b4 response_code = 200 uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?sort_by=id_ascending
104222025-05-08T22:12:51.937ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49336 req_id = 35405000-8fe4-4df5-b737-899d2da6754e roles = RoleSet { roles: {} } uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZGQ4MjYzNC0yODQ0LTRmOTctOTc0Ni0wNzA5OTA0NGMxOWQifX0%3D
104232025-05-08T22:12:51.937ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49336 req_id = 35405000-8fe4-4df5-b737-899d2da6754e resource = Database result = Ok(()) uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZGQ4MjYzNC0yODQ0LTRmOTctOTc0Ni0wNzA5OTA0NGMxOWQifX0%3D
104242025-05-08T22:12:51.939ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2794 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49336 req_id = 35405000-8fe4-4df5-b737-899d2da6754e response_code = 200 uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZGQ4MjYzNC0yODQ0LTRmOTctOTc0Ni0wNzA5OTA0NGMxOWQifX0%3D
104252025-05-08T22:12:51.940ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 314 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 62d36ad4-b14a-4eeb-adc2-2a64f3534556 response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
104262025-05-08T22:12:58.272ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 87 reason = Timeout
104272025-05-08T22:12:58.272ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
104282025-05-08T22:12:58.273ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
104292025-05-08T22:12:58.275ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
104302025-05-08T22:12:58.278ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
104312025-05-08T22:12:58.278ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
104322025-05-08T22:12:58.278ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
104332025-05-08T22:12:58.280ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
104342025-05-08T22:12:58.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
104352025-05-08T22:12:58.281ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
104362025-05-08T22:12:58.284ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
104372025-05-08T22:12:58.284ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
104382025-05-08T22:12:58.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
104392025-05-08T22:12:58.285ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 87 reason = Timeout
104402025-05-08T22:12:58.286ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
104412025-05-08T22:12:58.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
104422025-05-08T22:12:58.287ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
104432025-05-08T22:12:58.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
104442025-05-08T22:12:58.289ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.292945ms iteration = 87 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
104452025-05-08T22:12:58.293ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
104462025-05-08T22:12:58.293ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.716912ms iteration = 87 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
104472025-05-08T22:13:01.940ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 266 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 33ff4c0c-dcdb-4c4d-a96c-a6f700913b3a response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
104482025-05-08T22:13:06.933ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49336 req_id = 16e22dc8-bca7-414d-8ec4-a66c93d568f5 roles = RoleSet { roles: {} } uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?sort_by=id_ascending
104492025-05-08T22:13:06.934ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49336 req_id = 16e22dc8-bca7-414d-8ec4-a66c93d568f5 resource = Database result = Ok(()) uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?sort_by=id_ascending
104502025-05-08T22:13:06.937ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3377 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49336 req_id = 16e22dc8-bca7-414d-8ec4-a66c93d568f5 response_code = 200 uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?sort_by=id_ascending
104512025-05-08T22:13:06.937ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49336 req_id = 65f9a5a1-daf8-4fae-8ab4-ac74ebb24756 roles = RoleSet { roles: {} } uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZGQ4MjYzNC0yODQ0LTRmOTctOTc0Ni0wNzA5OTA0NGMxOWQifX0%3D
104522025-05-08T22:13:06.937ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49336 req_id = 65f9a5a1-daf8-4fae-8ab4-ac74ebb24756 resource = Database result = Ok(()) uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZGQ4MjYzNC0yODQ0LTRmOTctOTc0Ni0wNzA5OTA0NGMxOWQifX0%3D
104532025-05-08T22:13:06.939ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2451 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49336 req_id = 65f9a5a1-daf8-4fae-8ab4-ac74ebb24756 response_code = 200 uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZGQ4MjYzNC0yODQ0LTRmOTctOTc0Ni0wNzA5OTA0NGMxOWQifX0%3D
104542025-05-08T22:13:08.270ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 30 reason = Timeout
104552025-05-08T22:13:08.270ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
104562025-05-08T22:13:08.270ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 30 reason = Timeout
104572025-05-08T22:13:08.270ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
104582025-05-08T22:13:08.271ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
104592025-05-08T22:13:08.271ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
104602025-05-08T22:13:08.272ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 88 reason = Timeout
104612025-05-08T22:13:08.272ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 30 reason = Timeout
104622025-05-08T22:13:08.272ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
104632025-05-08T22:13:08.272ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
104642025-05-08T22:13:08.273ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1764, tv_nsec: 9872844 } })
104652025-05-08T22:13:08.273ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163
104662025-05-08T22:13:08.274ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 30 reason = Timeout
104672025-05-08T22:13:08.274ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
104682025-05-08T22:13:08.274ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1764, tv_nsec: 10989216 } })
104692025-05-08T22:13:08.274ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
104702025-05-08T22:13:08.274ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1163
104712025-05-08T22:13:08.275ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
104722025-05-08T22:13:08.276ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.581408ms iteration = 30 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
104732025-05-08T22:13:08.276ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
104742025-05-08T22:13:08.278ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
104752025-05-08T22:13:08.278ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
104762025-05-08T22:13:08.278ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
104772025-05-08T22:13:08.278ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
104782025-05-08T22:13:08.279ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
104792025-05-08T22:13:08.279ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
104802025-05-08T22:13:08.281ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
104812025-05-08T22:13:08.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
104822025-05-08T22:13:08.282ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
104832025-05-08T22:13:08.285ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
104842025-05-08T22:13:08.285ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
104852025-05-08T22:13:08.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
104862025-05-08T22:13:08.286ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 88 reason = Timeout
104872025-05-08T22:13:08.286ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
104882025-05-08T22:13:08.286ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 30 reason = Timeout
104892025-05-08T22:13:08.286ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
104902025-05-08T22:13:08.286ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 30 reason = Timeout
104912025-05-08T22:13:08.286ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
104922025-05-08T22:13:08.286ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
104932025-05-08T22:13:08.286ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 30 reason = Timeout
104942025-05-08T22:13:08.286ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
104952025-05-08T22:13:08.286ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 30 reason = Timeout
104962025-05-08T22:13:08.286ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 30 reason = Timeout
104972025-05-08T22:13:08.286ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
104982025-05-08T22:13:08.286ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
104992025-05-08T22:13:08.286ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
105002025-05-08T22:13:08.286ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 30 reason = Timeout
105012025-05-08T22:13:08.286ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
105022025-05-08T22:13:08.286ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 30 reason = Timeout
105032025-05-08T22:13:08.286ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 32 reason = Timeout
105042025-05-08T22:13:08.286ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
105052025-05-08T22:13:08.286ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
105062025-05-08T22:13:08.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
105072025-05-08T22:13:08.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
105082025-05-08T22:13:08.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
105092025-05-08T22:13:08.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
105102025-05-08T22:13:08.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
105112025-05-08T22:13:08.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
105122025-05-08T22:13:08.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
105132025-05-08T22:13:08.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
105142025-05-08T22:13:08.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
105152025-05-08T22:13:08.287ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 30 reason = Timeout
105162025-05-08T22:13:08.287ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
105172025-05-08T22:13:08.287ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 30 reason = Timeout
105182025-05-08T22:13:08.287ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 30 reason = Timeout
105192025-05-08T22:13:08.287ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
105202025-05-08T22:13:08.287ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
105212025-05-08T22:13:08.287ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 30 reason = Timeout
105222025-05-08T22:13:08.287ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 30 reason = Timeout
105232025-05-08T22:13:08.287ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
105242025-05-08T22:13:08.287ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
105252025-05-08T22:13:08.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
105262025-05-08T22:13:08.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
105272025-05-08T22:13:08.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
105282025-05-08T22:13:08.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
105292025-05-08T22:13:08.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
105302025-05-08T22:13:08.288ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
105312025-05-08T22:13:08.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
105322025-05-08T22:13:08.289ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105332025-05-08T22:13:08.289ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105342025-05-08T22:13:08.290ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105352025-05-08T22:13:08.290ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105362025-05-08T22:13:08.292ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
105372025-05-08T22:13:08.292ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
105382025-05-08T22:13:08.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
105392025-05-08T22:13:08.292ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
105402025-05-08T22:13:08.292ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
105412025-05-08T22:13:08.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
105422025-05-08T22:13:08.293ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
105432025-05-08T22:13:08.293ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
105442025-05-08T22:13:08.294ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
105452025-05-08T22:13:08.295ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
105462025-05-08T22:13:08.295ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 23.009446ms iteration = 88 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
105472025-05-08T22:13:08.295ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
105482025-05-08T22:13:08.295ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
105492025-05-08T22:13:08.296ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
105502025-05-08T22:13:08.298ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
105512025-05-08T22:13:08.299ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
105522025-05-08T22:13:08.301ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105532025-05-08T22:13:08.304ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
105542025-05-08T22:13:08.304ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
105552025-05-08T22:13:08.304ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
105562025-05-08T22:13:08.306ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
105572025-05-08T22:13:08.306ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
105582025-05-08T22:13:08.307ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1764, tv_nsec: 43985109 } })
105592025-05-08T22:13:08.308ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
105602025-05-08T22:13:08.308ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
105612025-05-08T22:13:08.308ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
105622025-05-08T22:13:08.311ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
105632025-05-08T22:13:08.312ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
105642025-05-08T22:13:08.326ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
105652025-05-08T22:13:08.327ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
105662025-05-08T22:13:08.332ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105672025-05-08T22:13:08.336ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
105682025-05-08T22:13:08.336ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
105692025-05-08T22:13:08.336ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
105702025-05-08T22:13:08.338ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105712025-05-08T22:13:08.339ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
105722025-05-08T22:13:08.340ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
105732025-05-08T22:13:08.342ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
105742025-05-08T22:13:08.342ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
105752025-05-08T22:13:08.342ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
105762025-05-08T22:13:08.352ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 65.100657ms iteration = 30 status = {"errors":[],"garbage_collect_requested":[]}
105772025-05-08T22:13:08.355ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
105782025-05-08T22:13:08.355ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
105792025-05-08T22:13:08.355ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
105802025-05-08T22:13:08.355ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
105812025-05-08T22:13:08.361ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 74.068622ms iteration = 30 status = {"errors":[],"requests_created_ok":[]}
105822025-05-08T22:13:08.363ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
105832025-05-08T22:13:08.363ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105842025-05-08T22:13:08.363ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 77.64176ms iteration = 88 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
105852025-05-08T22:13:08.364ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
105862025-05-08T22:13:08.365ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
105872025-05-08T22:13:08.366ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
105882025-05-08T22:13:08.366ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
105892025-05-08T22:13:08.366ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
105902025-05-08T22:13:08.366ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
105912025-05-08T22:13:08.369ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
105922025-05-08T22:13:08.369ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
105932025-05-08T22:13:08.369ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 82.521982ms iteration = 30 status = {"errors":[],"finish_invoked_ok":[]}
105942025-05-08T22:13:08.371ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
105952025-05-08T22:13:08.371ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
105962025-05-08T22:13:08.371ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "eba7b1fb-e370-47ec-8ec3-b7f8ceffe7f0", "content-length": "26", "date": "Thu, 08 May 2025 22:13:08 GMT"} })
105972025-05-08T22:13:08.371ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
105982025-05-08T22:13:08.372ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:552 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
105992025-05-08T22:13:08.372ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 86.705217ms iteration = 30 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
106002025-05-08T22:13:08.373ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
106012025-05-08T22:13:08.374ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 88.083852ms iteration = 30 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
106022025-05-08T22:13:08.375ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 88.225243ms iteration = 30 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
106032025-05-08T22:13:08.401ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e zone_address = fd00:1122:3344:101::2
106042025-05-08T22:13:08.401ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
106052025-05-08T22:13:08.401ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "98f40bdd-52f1-452e-baee-451d3bfeeab6", "content-length": "26", "date": "Thu, 08 May 2025 22:13:08 GMT"} })
106062025-05-08T22:13:08.401ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1174 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
106072025-05-08T22:13:08.402ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
106082025-05-08T22:13:08.402ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
106092025-05-08T22:13:08.402ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
106102025-05-08T22:13:08.402ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
106112025-05-08T22:13:08.403ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
106122025-05-08T22:13:08.406ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
106132025-05-08T22:13:08.408ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
106142025-05-08T22:13:08.408ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
106152025-05-08T22:13:08.408ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
106162025-05-08T22:13:08.458ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
106172025-05-08T22:13:08.458ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
106182025-05-08T22:13:08.458ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "18033017-aeca-4f44-8836-b295897d6efc", "content-length": "26", "date": "Thu, 08 May 2025 22:13:08 GMT"} })
106192025-05-08T22:13:08.458ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
106202025-05-08T22:13:08.460ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
106212025-05-08T22:13:08.474ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
106222025-05-08T22:13:08.474ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "915e833b-5194-4107-9d77-88723f9262d9", "content-length": "691", "date": "Thu, 08 May 2025 22:13:08 GMT"} })
106232025-05-08T22:13:08.474ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
106242025-05-08T22:13:08.474ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
106252025-05-08T22:13:08.474ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
106262025-05-08T22:13:08.477ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 191.079853ms iteration = 30 status = {}
106272025-05-08T22:13:08.530ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
106282025-05-08T22:13:08.530ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
106292025-05-08T22:13:08.530ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "afecefcc-288c-45c7-b150-9d5417a48d40", "content-length": "372", "date": "Thu, 08 May 2025 22:13:08 GMT"} })
106302025-05-08T22:13:08.530ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
106312025-05-08T22:13:08.531ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
106322025-05-08T22:13:08.538ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
106332025-05-08T22:13:08.538ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
106342025-05-08T22:13:08.541ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
106352025-05-08T22:13:08.542ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
106362025-05-08T22:13:08.543ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
106372025-05-08T22:13:08.548ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 2f2f2c32-8d58-4801-a669-9618eac0c3f7, lookup_type: ById(2f2f2c32-8d58-4801-a669-9618eac0c3f7) } result = Ok(())
106382025-05-08T22:13:08.548ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
106392025-05-08T22:13:08.549ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
106402025-05-08T22:13:08.551ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 265.428639ms iteration = 30 status = 0
106412025-05-08T22:13:08.571ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 300.410523ms iteration = 30 status = {}
106422025-05-08T22:13:08.580ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 293.362327ms iteration = 30 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
106432025-05-08T22:13:08.585ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
106442025-05-08T22:13:08.585ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
106452025-05-08T22:13:08.585ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 298.980848ms iteration = 30 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
106462025-05-08T22:13:08.587ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:362
106472025-05-08T22:13:08.587ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 313.455682ms iteration = 30 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
106482025-05-08T22:13:08.591ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
106492025-05-08T22:13:08.593ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
106502025-05-08T22:13:08.593ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
106512025-05-08T22:13:08.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
106522025-05-08T22:13:08.610ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
106532025-05-08T22:13:08.610ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
106542025-05-08T22:13:08.613ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
106552025-05-08T22:13:08.613ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
106562025-05-08T22:13:08.613ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "4e930025-43ed-4f69-b555-3ec36429c02c", "date": "Thu, 08 May 2025 22:13:08 GMT"} })
106572025-05-08T22:13:08.613ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
106582025-05-08T22:13:08.614ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
106592025-05-08T22:13:08.614ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 343.879338ms iteration = 30 status = null
106602025-05-08T22:13:08.616ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
106612025-05-08T22:13:08.616ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
106622025-05-08T22:13:08.624ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
106632025-05-08T22:13:08.625ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
106642025-05-08T22:13:08.629ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
106652025-05-08T22:13:08.629ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
106662025-05-08T22:13:08.629ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "0f2a8fa0-2ad1-414c-99c7-b33d251530fd", "date": "Thu, 08 May 2025 22:13:08 GMT"} })
106672025-05-08T22:13:08.630ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 344.581085ms iteration = 30 status = {}
106682025-05-08T22:13:08.692ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
106692025-05-08T22:13:08.692ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
106702025-05-08T22:13:08.694ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: cd0b930c-534b-46ad-ac3b-2eaa94b0bf2c, rack_id: 64f3d6f4-89b8-48aa-a622-b684e261667e, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(65f83a36-18fb-49b7-9914-87c8517d200e) } rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
106712025-05-08T22:13:08.694ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
106722025-05-08T22:13:08.694ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
106732025-05-08T22:13:08.705ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1833 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
106742025-05-08T22:13:08.705ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1860 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 65f83a36-18fb-49b7-9914-87c8517d200e
106752025-05-08T22:13:08.705ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
106762025-05-08T22:13:08.705ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
106772025-05-08T22:13:08.705ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
106782025-05-08T22:13:08.705ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
106792025-05-08T22:13:08.706ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
106802025-05-08T22:13:08.706ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2007 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
106812025-05-08T22:13:08.706ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
106822025-05-08T22:13:08.706ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2043 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
106832025-05-08T22:13:08.706ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
106842025-05-08T22:13:08.706ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
106852025-05-08T22:13:08.706ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
106862025-05-08T22:13:08.708ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 421.739811ms iteration = 32 status = {"physical_disks_added":0}
106872025-05-08T22:13:08.709ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
106882025-05-08T22:13:08.709ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
106892025-05-08T22:13:08.846ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "030133af-764b-48d9-82a8-8e14cfe00bf1", "date": "Thu, 08 May 2025 22:13:08 GMT"} })
106902025-05-08T22:13:08.846ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch = Switch0
106912025-05-08T22:13:08.846ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
106922025-05-08T22:13:08.846ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "d16a6b1f-19b8-4ed7-b922-5397a7a2e2f1", "content-length": "682", "date": "Thu, 08 May 2025 22:13:08 GMT"} })
106932025-05-08T22:13:08.846ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
106942025-05-08T22:13:08.846ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
106952025-05-08T22:13:08.850ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
106962025-05-08T22:13:08.850ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
106972025-05-08T22:13:08.854ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
106982025-05-08T22:13:08.854ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
106992025-05-08T22:13:08.856ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
107002025-05-08T22:13:08.856ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
107012025-05-08T22:13:08.863ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 577.002401ms iteration = 30 status = {}
107022025-05-08T22:13:11.941ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 260 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 6a4a87f3-e3b7-4daa-b760-91d65fc70f73 response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
107032025-05-08T22:13:18.272ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 89 reason = Timeout
107042025-05-08T22:13:18.272ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
107052025-05-08T22:13:18.273ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
107062025-05-08T22:13:18.274ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
107072025-05-08T22:13:18.277ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
107082025-05-08T22:13:18.277ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
107092025-05-08T22:13:18.277ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
107102025-05-08T22:13:18.279ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
107112025-05-08T22:13:18.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
107122025-05-08T22:13:18.281ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
107132025-05-08T22:13:18.283ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
107142025-05-08T22:13:18.283ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
107152025-05-08T22:13:18.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
107162025-05-08T22:13:18.286ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 89 reason = Timeout
107172025-05-08T22:13:18.286ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
107182025-05-08T22:13:18.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
107192025-05-08T22:13:18.294ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
107202025-05-08T22:13:18.294ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.50724ms iteration = 89 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
107212025-05-08T22:13:18.317ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
107222025-05-08T22:13:18.317ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
107232025-05-08T22:13:18.356ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 70.363922ms iteration = 89 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
107242025-05-08T22:13:21.934ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49336 req_id = f3f46f69-1a9a-4f24-abb3-c424c986e38a roles = RoleSet { roles: {} } uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?sort_by=id_ascending
107252025-05-08T22:13:21.934ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49336 req_id = f3f46f69-1a9a-4f24-abb3-c424c986e38a resource = Database result = Ok(()) uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?sort_by=id_ascending
107262025-05-08T22:13:21.937ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3175 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49336 req_id = f3f46f69-1a9a-4f24-abb3-c424c986e38a response_code = 200 uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?sort_by=id_ascending
107272025-05-08T22:13:21.937ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49336 req_id = b892debf-b7cc-4ca3-b77c-a3771aa68c75 roles = RoleSet { roles: {} } uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZGQ4MjYzNC0yODQ0LTRmOTctOTc0Ni0wNzA5OTA0NGMxOWQifX0%3D
107282025-05-08T22:13:21.937ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49336 req_id = b892debf-b7cc-4ca3-b77c-a3771aa68c75 resource = Database result = Ok(()) uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZGQ4MjYzNC0yODQ0LTRmOTctOTc0Ni0wNzA5OTA0NGMxOWQifX0%3D
107292025-05-08T22:13:21.939ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2118 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49336 req_id = b892debf-b7cc-4ca3-b77c-a3771aa68c75 response_code = 200 uri = /metrics/collectors/22ae79e2-3731-44c9-8346-78c5ddbfe577/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZGQ4MjYzNC0yODQ0LTRmOTctOTc0Ni0wNzA5OTA0NGMxOWQifX0%3D
107302025-05-08T22:13:21.941ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 230 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = b7e5815f-4a47-41b9-a43a-20ad52e504f9 response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
107312025-05-08T22:13:28.273ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 90 reason = Timeout
107322025-05-08T22:13:28.273ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
107332025-05-08T22:13:28.274ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
107342025-05-08T22:13:28.276ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
107352025-05-08T22:13:28.279ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
107362025-05-08T22:13:28.279ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
107372025-05-08T22:13:28.279ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
107382025-05-08T22:13:28.281ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
107392025-05-08T22:13:28.281ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
107402025-05-08T22:13:28.282ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
107412025-05-08T22:13:28.285ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
107422025-05-08T22:13:28.285ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
107432025-05-08T22:13:28.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
107442025-05-08T22:13:28.285ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 90 reason = Timeout
107452025-05-08T22:13:28.285ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
107462025-05-08T22:13:28.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
107472025-05-08T22:13:28.287ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
107482025-05-08T22:13:28.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
107492025-05-08T22:13:28.288ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.168994ms iteration = 90 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
107502025-05-08T22:13:28.297ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
107512025-05-08T22:13:28.297ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 23.469771ms iteration = 90 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
107522025-05-08T22:13:31.941ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 277 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 9d0d2680-5b0c-4f71-9cc1-3512c10a76a3 response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
107532025-05-08T22:13:38.271ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 31 reason = Timeout
107542025-05-08T22:13:38.271ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 18 reason = Timeout
107552025-05-08T22:13:38.271ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
107562025-05-08T22:13:38.271ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 16 reason = Timeout
107572025-05-08T22:13:38.271ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 31 reason = Timeout
107582025-05-08T22:13:38.271ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
107592025-05-08T22:13:38.271ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
107602025-05-08T22:13:38.271ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
107612025-05-08T22:13:38.271ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 16 reason = Timeout
107622025-05-08T22:13:38.271ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 16 reason = Timeout
107632025-05-08T22:13:38.271ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 16 reason = Timeout
107642025-05-08T22:13:38.271ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
107652025-05-08T22:13:38.271ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 18 reason = Timeout
107662025-05-08T22:13:38.271ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 16 reason = Timeout
107672025-05-08T22:13:38.271ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-05-08 22:03:38.271340471 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
107682025-05-08T22:13:38.271ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
107692025-05-08T22:13:38.271ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 16 reason = Timeout
107702025-05-08T22:13:38.271ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
107712025-05-08T22:13:38.271ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
107722025-05-08T22:13:38.271ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
107732025-05-08T22:13:38.271ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
107742025-05-08T22:13:38.271ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
107752025-05-08T22:13:38.271ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
107762025-05-08T22:13:38.271ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("228ed1c1-3cc1-4055-aae9-5595ad856b86.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("8b245480-f91d-406b-bb08-850e1c3e3fc5.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("c8f20f67-ea63-4dbc-b0f1-c30876daff82.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c8f20f67-ea63-4dbc-b0f1-c30876daff82.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 1794, tv_nsec: 7997784 } })
107772025-05-08T22:13:38.271ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("0ccabf41-81cd-45e7-bb26-335edaa2ee47.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("38a08dd6-6a75-4558-b4f8-df853f4d03d9.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("38a08dd6-6a75-4558-b4f8-df853f4d03d9.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::9))) }], valid_until: Instant { tv_sec: 1794, tv_nsec: 7994584 } })
107782025-05-08T22:13:38.272ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
107792025-05-08T22:13:38.272ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
107802025-05-08T22:13:38.272ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } dns_group = external
107812025-05-08T22:13:38.272ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.060091ms iteration = 16 status = {"addresses":["[fd00:1122:3344:101::8]:5353","[fd00:1122:3344:101::9]:5353"]}
107822025-05-08T22:13:38.272ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
107832025-05-08T22:13:38.272ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.141353ms iteration = 16 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:1::1]:5353"]}
107842025-05-08T22:13:38.273ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 91 reason = Timeout
107852025-05-08T22:13:38.273ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 17 reason = Timeout
107862025-05-08T22:13:38.273ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:85 reason = blueprint disabled target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2
107872025-05-08T22:13:38.273ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 31 reason = Timeout
107882025-05-08T22:13:38.273ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
107892025-05-08T22:13:38.273ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 6.411\u{b5}s iteration = 17 status = {"enabled":false,"target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2"}
107902025-05-08T22:13:38.273ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
107912025-05-08T22:13:38.274ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
107922025-05-08T22:13:38.274ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
107932025-05-08T22:13:38.274ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1794, tv_nsec: 10686743 } })
107942025-05-08T22:13:38.274ZDEBGnexus (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 7 reason = Timeout
107952025-05-08T22:13:38.274ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 31 reason = Timeout
107962025-05-08T22:13:38.274ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
107972025-05-08T22:13:38.274ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
107982025-05-08T22:13:38.274ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1794, tv_nsec: 10892365 } })
107992025-05-08T22:13:38.274ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1163
108002025-05-08T22:13:38.275ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
108012025-05-08T22:13:38.275ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
108022025-05-08T22:13:38.275ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
108032025-05-08T22:13:38.275ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163
108042025-05-08T22:13:38.275ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
108052025-05-08T22:13:38.275ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
108062025-05-08T22:13:38.275ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
108072025-05-08T22:13:38.275ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
108082025-05-08T22:13:38.275ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
108092025-05-08T22:13:38.275ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-05-08 22:03:38.271340471 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
108102025-05-08T22:13:38.275ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 4.131355ms iteration = 16 status = {"expiration":"2025-05-08T22:03:38.271340471Z","pruned":[]}
108112025-05-08T22:13:38.275ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
108122025-05-08T22:13:38.275ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
108132025-05-08T22:13:38.275ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
108142025-05-08T22:13:38.275ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
108152025-05-08T22:13:38.276ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
108162025-05-08T22:13:38.278ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
108172025-05-08T22:13:38.278ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
108182025-05-08T22:13:38.278ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
108192025-05-08T22:13:38.279ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 5.988533ms iteration = 31 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
108202025-05-08T22:13:38.279ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
108212025-05-08T22:13:38.281ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
108222025-05-08T22:13:38.281ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
108232025-05-08T22:13:38.281ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
108242025-05-08T22:13:38.283ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
108252025-05-08T22:13:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
108262025-05-08T22:13:38.284ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
108272025-05-08T22:13:38.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
108282025-05-08T22:13:38.285ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
108292025-05-08T22:13:38.285ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
108302025-05-08T22:13:38.285ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
108312025-05-08T22:13:38.285ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
108322025-05-08T22:13:38.286ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 91 reason = Timeout
108332025-05-08T22:13:38.286ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 31 reason = Timeout
108342025-05-08T22:13:38.286ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
108352025-05-08T22:13:38.286ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
108362025-05-08T22:13:38.286ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 31 reason = Timeout
108372025-05-08T22:13:38.286ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
108382025-05-08T22:13:38.286ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 31 reason = Timeout
108392025-05-08T22:13:38.286ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
108402025-05-08T22:13:38.286ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
108412025-05-08T22:13:38.286ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 31 reason = Timeout
108422025-05-08T22:13:38.286ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
108432025-05-08T22:13:38.286ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 31 reason = Timeout
108442025-05-08T22:13:38.286ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
108452025-05-08T22:13:38.286ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 31 reason = Timeout
108462025-05-08T22:13:38.286ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
108472025-05-08T22:13:38.286ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 16 reason = Timeout
108482025-05-08T22:13:38.286ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:53
108492025-05-08T22:13:38.286ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
108502025-05-08T22:13:38.286ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 6 reason = Timeout
108512025-05-08T22:13:38.286ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
108522025-05-08T22:13:38.286ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 33 reason = Timeout
108532025-05-08T22:13:38.286ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
108542025-05-08T22:13:38.286ZDEBGnexus (BackgroundTasks): activating background_task = service_firewall_rule_propagation iteration = 4 reason = Timeout
108552025-05-08T22:13:38.286ZDEBGnexus (service-firewall-rule-progator): starting background task for service firewall rule propagation background_task = service_firewall_rule_propagation
108562025-05-08T22:13:38.286ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
108572025-05-08T22:13:38.286ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 31 reason = Timeout
108582025-05-08T22:13:38.286ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
108592025-05-08T22:13:38.286ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
108602025-05-08T22:13:38.286ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 16 reason = Timeout
108612025-05-08T22:13:38.286ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
108622025-05-08T22:13:38.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
108632025-05-08T22:13:38.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
108642025-05-08T22:13:38.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
108652025-05-08T22:13:38.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
108662025-05-08T22:13:38.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
108672025-05-08T22:13:38.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
108682025-05-08T22:13:38.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
108692025-05-08T22:13:38.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
108702025-05-08T22:13:38.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
108712025-05-08T22:13:38.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
108722025-05-08T22:13:38.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
108732025-05-08T22:13:38.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
108742025-05-08T22:13:38.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
108752025-05-08T22:13:38.289ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 16 reason = Timeout
108762025-05-08T22:13:38.289ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
108772025-05-08T22:13:38.289ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
108782025-05-08T22:13:38.289ZDEBGnexus (BackgroundTasks): activating background_task = webhook_deliverator iteration = 16 reason = Timeout
108792025-05-08T22:13:38.289ZDEBGnexus (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} }
108802025-05-08T22:13:38.289ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
108812025-05-08T22:13:38.289ZDEBGnexus (BackgroundTasks): activating background_task = tuf_artifact_replication iteration = 4 reason = Timeout
108822025-05-08T22:13:38.289ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
108832025-05-08T22:13:38.289ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 31 reason = Timeout
108842025-05-08T22:13:38.289ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
108852025-05-08T22:13:38.289ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 16 reason = Timeout
108862025-05-08T22:13:38.289ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
108872025-05-08T22:13:38.289ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 31 reason = Timeout
108882025-05-08T22:13:38.289ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
108892025-05-08T22:13:38.289ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database result = Ok(())
108902025-05-08T22:13:38.289ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 31 reason = Timeout
108912025-05-08T22:13:38.289ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
108922025-05-08T22:13:38.289ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
108932025-05-08T22:13:38.289ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 31 reason = Timeout
108942025-05-08T22:13:38.289ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
108952025-05-08T22:13:38.289ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
108962025-05-08T22:13:38.289ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 31 reason = Timeout
108972025-05-08T22:13:38.289ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
108982025-05-08T22:13:38.290ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
108992025-05-08T22:13:38.290ZDEBGnexus (BackgroundTasks): activating background_task = webhook_dispatcher iteration = 16 reason = Timeout
109002025-05-08T22:13:38.290ZDEBGnexus (BackgroundTasks): roles background_task = webhook_dispatcher roles = RoleSet { roles: {} }
109012025-05-08T22:13:38.290ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
109022025-05-08T22:13:38.290ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
109032025-05-08T22:13:38.290ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
109042025-05-08T22:13:38.290ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database result = Ok(())
109052025-05-08T22:13:38.290ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
109062025-05-08T22:13:38.292ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
109072025-05-08T22:13:38.292ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
109082025-05-08T22:13:38.292ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
109092025-05-08T22:13:38.295ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109102025-05-08T22:13:38.296ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
109112025-05-08T22:13:38.296ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
109122025-05-08T22:13:38.299ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
109132025-05-08T22:13:38.299ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
109142025-05-08T22:13:38.299ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
109152025-05-08T22:13:38.301ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
109162025-05-08T22:13:38.302ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
109172025-05-08T22:13:38.303ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109182025-05-08T22:13:38.304ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109192025-05-08T22:13:38.304ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109202025-05-08T22:13:38.304ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109212025-05-08T22:13:38.307ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
109222025-05-08T22:13:38.307ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
109232025-05-08T22:13:38.307ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
109242025-05-08T22:13:38.307ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
109252025-05-08T22:13:38.307ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
109262025-05-08T22:13:38.308ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
109272025-05-08T22:13:38.309ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109282025-05-08T22:13:38.311ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
109292025-05-08T22:13:38.311ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
109302025-05-08T22:13:38.311ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Inventory result = Ok(())
109312025-05-08T22:13:38.311ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
109322025-05-08T22:13:38.311ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
109332025-05-08T22:13:38.311ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
109342025-05-08T22:13:38.311ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
109352025-05-08T22:13:38.311ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
109362025-05-08T22:13:38.311ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
109372025-05-08T22:13:38.311ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
109382025-05-08T22:13:38.313ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
109392025-05-08T22:13:38.313ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
109402025-05-08T22:13:38.313ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
109412025-05-08T22:13:38.325ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
109422025-05-08T22:13:38.326ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
109432025-05-08T22:13:38.332ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109442025-05-08T22:13:38.332ZDEBGnexus (BackgroundTasks): activation complete background_task = webhook_deliverator elapsed = 43.571106ms iteration = 16 status = {"by_rx":{},"error":null}
109452025-05-08T22:13:38.332ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 44.157703ms iteration = 16 status = {"errors":[],"found_port_ok":[]}
109462025-05-08T22:13:38.334ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
109472025-05-08T22:13:38.334ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
109482025-05-08T22:13:38.335ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
109492025-05-08T22:13:38.342ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109502025-05-08T22:13:38.345ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109512025-05-08T22:13:38.348ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
109522025-05-08T22:13:38.348ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
109532025-05-08T22:13:38.348ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109542025-05-08T22:13:38.348ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
109552025-05-08T22:13:38.351ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
109562025-05-08T22:13:38.351ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
109572025-05-08T22:13:38.351ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
109582025-05-08T22:13:38.352ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
109592025-05-08T22:13:38.352ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
109602025-05-08T22:13:38.353ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
109612025-05-08T22:13:38.356ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109622025-05-08T22:13:38.358ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
109632025-05-08T22:13:38.358ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
109642025-05-08T22:13:38.358ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
109652025-05-08T22:13:38.358ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
109662025-05-08T22:13:38.358ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
109672025-05-08T22:13:38.359ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
109682025-05-08T22:13:38.359ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
109692025-05-08T22:13:38.360ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
109702025-05-08T22:13:38.360ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
109712025-05-08T22:13:38.360ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
109722025-05-08T22:13:38.361ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109732025-05-08T22:13:38.362ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
109742025-05-08T22:13:38.362ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
109752025-05-08T22:13:38.362ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
109762025-05-08T22:13:38.362ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
109772025-05-08T22:13:38.362ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109782025-05-08T22:13:38.362ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
109792025-05-08T22:13:38.363ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109802025-05-08T22:13:38.364ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
109812025-05-08T22:13:38.364ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
109822025-05-08T22:13:38.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
109832025-05-08T22:13:38.365ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
109842025-05-08T22:13:38.365ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
109852025-05-08T22:13:38.365ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
109862025-05-08T22:13:38.366ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
109872025-05-08T22:13:38.366ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
109882025-05-08T22:13:38.367ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
109892025-05-08T22:13:38.367ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
109902025-05-08T22:13:38.367ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
109912025-05-08T22:13:38.367ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
109922025-05-08T22:13:38.367ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
109932025-05-08T22:13:38.368ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109942025-05-08T22:13:38.368ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
109952025-05-08T22:13:38.368ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
109962025-05-08T22:13:38.369ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
109972025-05-08T22:13:38.369ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
109982025-05-08T22:13:38.371ZDEBGnexus (BackgroundTasks): roles background_task = webhook_dispatcher roles = RoleSet { roles: {} }
109992025-05-08T22:13:38.371ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
110002025-05-08T22:13:38.371ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
110012025-05-08T22:13:38.371ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database result = Ok(())
110022025-05-08T22:13:38.371ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
110032025-05-08T22:13:38.371ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
110042025-05-08T22:13:38.372ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
110052025-05-08T22:13:38.374ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
110062025-05-08T22:13:38.375ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
110072025-05-08T22:13:38.375ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
110082025-05-08T22:13:38.375ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
110092025-05-08T22:13:38.375ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
110102025-05-08T22:13:38.375ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
110112025-05-08T22:13:38.375ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
110122025-05-08T22:13:38.379ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
110132025-05-08T22:13:38.379ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
110142025-05-08T22:13:38.379ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
110152025-05-08T22:13:38.380ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
110162025-05-08T22:13:38.380ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
110172025-05-08T22:13:38.380ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
110182025-05-08T22:13:38.381ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
110192025-05-08T22:13:38.381ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
110202025-05-08T22:13:38.381ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
110212025-05-08T22:13:38.384ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-2f2f2c32-8d58-4801-a669-9618eac0c3f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1794, tv_nsec: 120456357 } })
110222025-05-08T22:13:38.386ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
110232025-05-08T22:13:38.388ZDEBGnexus (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 113.739736ms iteration = 7 status = {"nsuccess":5}
110242025-05-08T22:13:38.389ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
110252025-05-08T22:13:38.389ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
110262025-05-08T22:13:38.391ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
110272025-05-08T22:13:38.391ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
110282025-05-08T22:13:38.393ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 104.355736ms iteration = 31 status = {"errors":[],"garbage_collect_requested":[]}
110292025-05-08T22:13:38.396ZDEBGnexus (BackgroundTasks): activation complete background_task = webhook_dispatcher elapsed = 106.883953ms iteration = 16 status = {"dispatched":[],"errors":[],"glob_version":"140.0.0","globs_reprocessed":{},"no_receivers":[]}
110302025-05-08T22:13:38.402ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
110312025-05-08T22:13:38.406ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
110322025-05-08T22:13:38.406ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
110332025-05-08T22:13:38.407ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
110342025-05-08T22:13:38.411ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 121.947574ms iteration = 31 status = {"errors":[],"finish_invoked_ok":[]}
110352025-05-08T22:13:38.412ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
110362025-05-08T22:13:38.414ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 127.869727ms iteration = 91 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
110372025-05-08T22:13:38.416ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
110382025-05-08T22:13:38.417ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 127.779157ms iteration = 31 status = {"errors":[],"requests_created_ok":[]}
110392025-05-08T22:13:38.417ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
110402025-05-08T22:13:38.418ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
110412025-05-08T22:13:38.418ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
110422025-05-08T22:13:38.418ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
110432025-05-08T22:13:38.418ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:137 found = 0 not_ready_to_be_deleted = 0
110442025-05-08T22:13:38.418ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
110452025-05-08T22:13:38.418ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
110462025-05-08T22:13:38.418ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 131.90445ms iteration = 16 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
110472025-05-08T22:13:38.418ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 129.0517ms iteration = 16 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
110482025-05-08T22:13:38.418ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
110492025-05-08T22:13:38.420ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
110502025-05-08T22:13:38.420ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
110512025-05-08T22:13:38.420ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
110522025-05-08T22:13:38.420ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
110532025-05-08T22:13:38.423ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
110542025-05-08T22:13:38.423ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
110552025-05-08T22:13:38.428ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:552 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
110562025-05-08T22:13:38.428ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 142.756656ms iteration = 31 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
110572025-05-08T22:13:38.429ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
110582025-05-08T22:13:38.429ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
110592025-05-08T22:13:38.430ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "b814fdd4-6737-4933-a91b-76df113f83e0", "content-length": "26", "date": "Thu, 08 May 2025 22:13:38 GMT"} })
110602025-05-08T22:13:38.430ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
110612025-05-08T22:13:38.431ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
110622025-05-08T22:13:38.431ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
110632025-05-08T22:13:38.432ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
110642025-05-08T22:13:38.433ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
110652025-05-08T22:13:38.439ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
110662025-05-08T22:13:38.440ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
110672025-05-08T22:13:38.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
110682025-05-08T22:13:38.441ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
110692025-05-08T22:13:38.441ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
110702025-05-08T22:13:38.442ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
110712025-05-08T22:13:38.442ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
110722025-05-08T22:13:38.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
110732025-05-08T22:13:38.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
110742025-05-08T22:13:38.442ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
110752025-05-08T22:13:38.442ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 169.943217ms iteration = 91 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
110762025-05-08T22:13:38.444ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
110772025-05-08T22:13:38.445ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
110782025-05-08T22:13:38.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
110792025-05-08T22:13:38.445ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
110802025-05-08T22:13:38.446ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
110812025-05-08T22:13:38.448ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
110822025-05-08T22:13:38.448ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
110832025-05-08T22:13:38.448ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
110842025-05-08T22:13:38.448ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
110852025-05-08T22:13:38.449ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
110862025-05-08T22:13:38.457ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
110872025-05-08T22:13:38.457ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
110882025-05-08T22:13:38.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
110892025-05-08T22:13:38.459ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 2f2f2c32-8d58-4801-a669-9618eac0c3f7, lookup_type: ById(2f2f2c32-8d58-4801-a669-9618eac0c3f7) } result = Ok(())
110902025-05-08T22:13:38.459ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
110912025-05-08T22:13:38.462ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
110922025-05-08T22:13:38.467ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:362
110932025-05-08T22:13:38.467ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 192.342667ms iteration = 31 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
110942025-05-08T22:13:38.471ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
110952025-05-08T22:13:38.472ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
110962025-05-08T22:13:38.475ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
110972025-05-08T22:13:38.476ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
110982025-05-08T22:13:38.476ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 190.562117ms iteration = 31 status = 0
110992025-05-08T22:13:38.476ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
111002025-05-08T22:13:38.479ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
111012025-05-08T22:13:38.479ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
111022025-05-08T22:13:38.479ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 193.229386ms iteration = 31 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
111032025-05-08T22:13:38.479ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
111042025-05-08T22:13:38.480ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
111052025-05-08T22:13:38.480ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
111062025-05-08T22:13:38.480ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 194.229137ms iteration = 31 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
111072025-05-08T22:13:38.481ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
111082025-05-08T22:13:38.481ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
111092025-05-08T22:13:38.484ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
111102025-05-08T22:13:38.484ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
111112025-05-08T22:13:38.485ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 196.113157ms iteration = 31 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
111122025-05-08T22:13:38.486ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
111132025-05-08T22:13:38.487ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
111142025-05-08T22:13:38.489ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
111152025-05-08T22:13:38.489ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 7586992a-6fc5-4073-8962-728778f16aa5 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
111162025-05-08T22:13:38.489ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
111172025-05-08T22:13:38.489ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
111182025-05-08T22:13:38.489ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
111192025-05-08T22:13:38.489ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
111202025-05-08T22:13:38.490ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
111212025-05-08T22:13:38.491ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
111222025-05-08T22:13:38.491ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
111232025-05-08T22:13:38.492ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
111242025-05-08T22:13:38.492ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
111252025-05-08T22:13:38.493ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = d8128696-db5a-4b88-914a-69b28494e50e dns_zone_name = oxide.test found_so_far = 0 version = 2
111262025-05-08T22:13:38.493ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
111272025-05-08T22:13:38.493ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
111282025-05-08T22:13:38.496ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
111292025-05-08T22:13:38.496ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
111302025-05-08T22:13:38.496ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
111312025-05-08T22:13:38.496ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-05-08T21:58:35.560911Z, time_modified: 2025-05-08T21:58:35.560911Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, db_silo: Silo { identity: SiloIdentity { id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-05-08T21:59:21.709488Z, time_modified: 2025-05-08T21:59:21.709488Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, db_silo: Silo { identity: SiloIdentity { id: 7137ee58-6aa7-4199-a248-feadf3b8bafb, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-05-08T21:59:21.709488Z, time_modified: 2025-05-08T21:59:21.709488Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47" }] }) }
111322025-05-08T22:13:38.496ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 225.696014ms iteration = 16 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"7137ee58-6aa7-4199-a248-feadf3b8bafb","tls_certs":["37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47"]}},"default_endpoint":{"silo_id":"7137ee58-6aa7-4199-a248-feadf3b8bafb","tls_certs":["37f9c8617947c893f8a2fcb2163a83145c9530e8959a1d2b94965fc9d1178e47"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
111332025-05-08T22:13:38.498ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
111342025-05-08T22:13:38.498ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
111352025-05-08T22:13:38.498ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
111362025-05-08T22:13:38.500ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
111372025-05-08T22:13:38.502ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
111382025-05-08T22:13:38.502ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 7586992a-6fc5-4073-8962-728778f16aa5 dns_zone_name = control-plane.oxide.internal found_so_far = 55 version = 1
111392025-05-08T22:13:38.502ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
111402025-05-08T22:13:38.502ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-05-08 21:59:21.182334 UTC dns_group = internal generation = 1
111412025-05-08T22:13:38.502ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 231.183092ms iteration = 16 status = {"generation":1}
111422025-05-08T22:13:38.502ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
111432025-05-08T22:13:38.504ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
111442025-05-08T22:13:38.504ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
111452025-05-08T22:13:38.504ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
111462025-05-08T22:13:38.505ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
111472025-05-08T22:13:38.505ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
111482025-05-08T22:13:38.507ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = d8128696-db5a-4b88-914a-69b28494e50e dns_zone_name = oxide.test found_so_far = 1 version = 2
111492025-05-08T22:13:38.507ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
111502025-05-08T22:13:38.507ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-05-08 21:59:21.759368 UTC dns_group = external generation = 2
111512025-05-08T22:13:38.507ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 236.185376ms iteration = 16 status = {"generation":2}
111522025-05-08T22:13:38.508ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
111532025-05-08T22:13:38.508ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
111542025-05-08T22:13:38.511ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
111552025-05-08T22:13:38.511ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
111562025-05-08T22:13:38.512ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
111572025-05-08T22:13:38.512ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
111582025-05-08T22:13:38.513ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
111592025-05-08T22:13:38.513ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
111602025-05-08T22:13:38.514ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
111612025-05-08T22:13:38.515ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
111622025-05-08T22:13:38.515ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
111632025-05-08T22:13:38.515ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
111642025-05-08T22:13:38.515ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
111652025-05-08T22:13:38.519ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
111662025-05-08T22:13:38.520ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
111672025-05-08T22:13:38.521ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
111682025-05-08T22:13:38.521ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
111692025-05-08T22:13:38.542ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
111702025-05-08T22:13:38.549ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 262.628839ms iteration = 33 status = {"physical_disks_added":0}
111712025-05-08T22:13:38.564ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 275.032781ms iteration = 31 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
111722025-05-08T22:13:38.592ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171 zone_address = fd00:1122:3344:101::2
111732025-05-08T22:13:38.592ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
111742025-05-08T22:13:38.593ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "903797fd-5ecd-4795-a6a5-2690dc62159f", "content-length": "26", "date": "Thu, 08 May 2025 22:13:38 GMT"} })
111752025-05-08T22:13:38.593ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1174 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
111762025-05-08T22:13:38.594ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
111772025-05-08T22:13:38.650ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e zone_address = fd00:1122:3344:101::2
111782025-05-08T22:13:38.651ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
111792025-05-08T22:13:38.651ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "3384885d-d674-42fb-9768-f6f431e92984", "content-length": "26", "date": "Thu, 08 May 2025 22:13:38 GMT"} })
111802025-05-08T22:13:38.651ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1174 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
111812025-05-08T22:13:38.655ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1208 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
111822025-05-08T22:13:38.664ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 393.011493ms iteration = 31 status = {}
111832025-05-08T22:13:38.738ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
111842025-05-08T22:13:38.738ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
111852025-05-08T22:13:38.738ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9a47849-0ef4-4cdc-9cd2-351387c45121", "content-length": "372", "date": "Thu, 08 May 2025 22:13:38 GMT"} })
111862025-05-08T22:13:38.738ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
111872025-05-08T22:13:38.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
111882025-05-08T22:13:38.739ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
111892025-05-08T22:13:38.741ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
111902025-05-08T22:13:38.741ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
111912025-05-08T22:13:38.744ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
111922025-05-08T22:13:38.744ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
111932025-05-08T22:13:38.744ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
111942025-05-08T22:13:38.746ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
111952025-05-08T22:13:38.746ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
111962025-05-08T22:13:38.746ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
111972025-05-08T22:13:38.748ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
111982025-05-08T22:13:38.748ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
111992025-05-08T22:13:38.748ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
112002025-05-08T22:13:38.748ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
112012025-05-08T22:13:38.749ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
112022025-05-08T22:13:38.749ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
112032025-05-08T22:13:38.749ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
112042025-05-08T22:13:38.750ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
112052025-05-08T22:13:38.750ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
112062025-05-08T22:13:38.750ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
112072025-05-08T22:13:38.754ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
112082025-05-08T22:13:38.754ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
112092025-05-08T22:13:38.754ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
112102025-05-08T22:13:38.754ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
112112025-05-08T22:13:38.754ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
112122025-05-08T22:13:38.754ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
112132025-05-08T22:13:38.754ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
112142025-05-08T22:13:38.754ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
112152025-05-08T22:13:38.755ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
112162025-05-08T22:13:38.756ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
112172025-05-08T22:13:38.756ZINFOnexus (BackgroundTasks): ensured all Crucible datasets present in inventory have database records background_task = blueprint_rendezvous file = nexus/reconfigurator/rendezvous/src/crucible_dataset.rs:113 num_already_exist = 9 num_inserted = 0 num_not_in_inventory = 0
112182025-05-08T22:13:38.756ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 470.617423ms iteration = 6 status = {"blueprint_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","inventory_collection_id":"b163257d-77a0-4b31-9b71-80dbd869fa80","stats":{"crucible_dataset":{"num_already_exist":9,"num_inserted":0,"num_not_in_inventory":0},"debug_dataset":{"num_already_exist":9,"num_already_tombstoned":0,"num_inserted":0,"num_not_in_inventory":0,"num_tombstoned":0}}}
112192025-05-08T22:13:38.757ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
112202025-05-08T22:13:38.757ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
112212025-05-08T22:13:38.758ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
112222025-05-08T22:13:38.760ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
112232025-05-08T22:13:38.761ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
112242025-05-08T22:13:38.761ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
112252025-05-08T22:13:38.761ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "d7c412b4-81e8-4563-9e86-caf8b972f50a", "date": "Thu, 08 May 2025 22:13:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
112262025-05-08T22:13:38.761ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
112272025-05-08T22:13:38.763ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
112282025-05-08T22:13:38.763ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "220ea415-5ec8-4f96-b373-d2e0814e280f", "date": "Thu, 08 May 2025 22:13:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
112292025-05-08T22:13:38.763ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
112302025-05-08T22:13:38.763ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
112312025-05-08T22:13:38.765ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
112322025-05-08T22:13:38.765ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::8]:5353, [fd00:1122:3344:101::9]:5353] }
112332025-05-08T22:13:38.765ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 493.821122ms iteration = 18 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
112342025-05-08T22:13:38.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
112352025-05-08T22:13:38.767ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
112362025-05-08T22:13:38.767ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
112372025-05-08T22:13:38.768ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
112382025-05-08T22:13:38.768ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
112392025-05-08T22:13:38.774ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
112402025-05-08T22:13:38.775ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
112412025-05-08T22:13:38.776ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
112422025-05-08T22:13:38.778ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
112432025-05-08T22:13:38.778ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
112442025-05-08T22:13:38.778ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
112452025-05-08T22:13:38.780ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
112462025-05-08T22:13:38.780ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
112472025-05-08T22:13:38.782ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
112482025-05-08T22:13:38.784ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
112492025-05-08T22:13:38.784ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
112502025-05-08T22:13:38.785ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
112512025-05-08T22:13:38.785ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
112522025-05-08T22:13:38.785ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
112532025-05-08T22:13:38.787ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = tuf_artifact_replication body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/artifacts-config
112542025-05-08T22:13:38.787ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = tuf_artifact_replication result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/artifacts-config", status: 204, headers: {"x-request-id": "a3969399-aea6-4ede-853c-3bcd45c05bac", "date": "Thu, 08 May 2025 22:13:38 GMT"} })
112552025-05-08T22:13:38.787ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = tuf_artifact_replication body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/artifacts
112562025-05-08T22:13:38.787ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = tuf_artifact_replication result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/artifacts", status: 200, headers: {"content-type": "application/json", "x-request-id": "4119be57-5db7-49ef-b031-a37d02cd96b9", "content-length": "26", "date": "Thu, 08 May 2025 22:13:38 GMT"} })
112572025-05-08T22:13:38.787ZINFOnexus (BackgroundTasks): Successfully got artifact list background_task = tuf_artifact_replication file = nexus/src/app/background/tasks/tuf_artifact_replication.rs:673 sled = http://[fd00:1122:3344:101::1]:12345
112582025-05-08T22:13:38.787ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 516.748667ms iteration = 31 status = null
112592025-05-08T22:13:38.789ZDEBGnexus (BackgroundTasks): activation complete background_task = tuf_artifact_replication elapsed = 500.721275ms iteration = 4 status = {"generation":1,"last_run_counters":{"copy_err":0,"copy_ok":0,"list_err":0,"list_ok":1,"put_config_err":0,"put_config_ok":1,"put_err":0,"put_ok":0},"lifetime_counters":{"copy_err":0,"copy_ok":0,"list_err":0,"list_ok":3,"put_config_err":0,"put_config_ok":3,"put_err":0,"put_ok":0},"local_repos":0,"request_debug_ringbuf":[{"generation":1,"operation":"put_config","target_sled":"2f2f2c32-8d58-4801-a669-9618eac0c3f7","time":"2025-05-08T22:03:39.017442287Z"},{"operation":"list","target_sled":"2f2f2c32-8d58-4801-a669-9618eac0c3f7","time":"2025-05-08T22:03:39.022391931Z"},{"generation":1,"operation":"put_config","target_sled":"2f2f2c32-8d58-4801-a669-9618eac0c3f7","time":"2025-05-08T22:08:39.002705581Z"},{"operation":"list","target_sled":"2f2f2c32-8d58-4801-a669-9618eac0c3f7","time":"2025-05-08T22:08:39.003341248Z"},{"generation":1,"operation":"put_config","target_sled":"2f2f2c32-8d58-4801-a669-9618eac0c3f7","time":"2025-05-08T22:13:38.787538852Z"},{"operation":"list","target_sled":"2f2f2c32-8d58-4801-a669-9618eac0c3f7","time":"2025-05-08T22:13:38.787780885Z"}]}
112602025-05-08T22:13:38.790ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
112612025-05-08T22:13:38.790ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
112622025-05-08T22:13:38.791ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "4dcf0f83-21e0-4a3a-9df2-afc7b5cb4c7a", "date": "Thu, 08 May 2025 22:13:38 GMT"} })
112632025-05-08T22:13:38.791ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
112642025-05-08T22:13:38.791ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
112652025-05-08T22:13:38.793ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
112662025-05-08T22:13:38.794ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
112672025-05-08T22:13:38.797ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
112682025-05-08T22:13:38.797ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
112692025-05-08T22:13:38.801ZINFOnexus (BackgroundTasks): VPC route manager sled 2f2f2c32-8d58-4801-a669-9618eac0c3f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
112702025-05-08T22:13:38.801ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
112712025-05-08T22:13:38.801ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "2cf26c3d-23c1-4462-bf9e-eac71e73f699", "date": "Thu, 08 May 2025 22:13:38 GMT"} })
112722025-05-08T22:13:38.803ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 517.412714ms iteration = 31 status = {}
112732025-05-08T22:13:38.804ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
112742025-05-08T22:13:38.804ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
112752025-05-08T22:13:38.805ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
112762025-05-08T22:13:38.805ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
112772025-05-08T22:13:38.805ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
112782025-05-08T22:13:38.807ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
112792025-05-08T22:13:38.817ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
112802025-05-08T22:13:38.817ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
112812025-05-08T22:13:38.817ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
112822025-05-08T22:13:38.818ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
112832025-05-08T22:13:38.819ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
112842025-05-08T22:13:38.821ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
112852025-05-08T22:13:38.821ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
112862025-05-08T22:13:38.823ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
112872025-05-08T22:13:38.831ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
112882025-05-08T22:13:38.831ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
112892025-05-08T22:13:38.831ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
112902025-05-08T22:13:38.841ZDEBGnexus (service-firewall-rule-progator): resolved names for firewall rules background_task = service_firewall_rule_propagation instance_interfaces = {} subnet_interfaces = {Name("external-dns"): [NetworkInterface { id: 517232ae-3fd2-40fd-801d-796bf5ecd14e, kind: Service { id: 0ccabf41-81cd-45e7-bb26-335edaa2ee47 }, name: Name("external-dns-0ccabf41-81cd-45e7-bb26-335edaa2ee47"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 224, 221])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: b845b0e9-661e-4fa1-9292-0c7dce84b88d, kind: Service { id: 38a08dd6-6a75-4558-b4f8-df853f4d03d9 }, name: Name("external-dns-38a08dd6-6a75-4558-b4f8-df853f4d03d9"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 179, 43])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], Name("nexus"): [NetworkInterface { id: 559b827c-9574-475a-93d6-1045f64398e6, kind: Service { id: 85a39cf9-9419-458f-af2e-c8b40d8f5f79 }, name: Name("nexus-85a39cf9-9419-458f-af2e-c8b40d8f5f79"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 232, 9])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 715a8bac-a423-4460-aac2-b8c76af6bb34, kind: Service { id: ddd82634-2844-4f97-9746-07099044c19d }, name: Name("nexus-ddd82634-2844-4f97-9746-07099044c19d"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 227, 84])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 76144c16-58bd-4abd-b758-565497d9f6b8, kind: Service { id: 53ef6679-65e8-4486-9401-d6fa4ff2e8e4 }, name: Name("nexus-53ef6679-65e8-4486-9401-d6fa4ff2e8e4"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 174, 62])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }]} subnet_networks = {Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })], Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })]} vpc_interfaces = {}
112912025-05-08T22:13:38.841ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
112922025-05-08T22:13:38.841ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
112932025-05-08T22:13:38.843ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
112942025-05-08T22:13:38.846ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Fleet result = Ok(())
112952025-05-08T22:13:38.846ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
112962025-05-08T22:13:38.846ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
112972025-05-08T22:13:38.848ZDEBGnexus (service-firewall-rule-progator): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified. background_task = service_firewall_rule_propagation
112982025-05-08T22:13:38.848ZDEBGnexus (service-firewall-rule-progator): resolved firewall rules for sled agents background_task = service_firewall_rule_propagation sled_agent_rules = [ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("53")]), filter_protocols: Some([Udp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 517232ae-3fd2-40fd-801d-796bf5ecd14e, kind: Service { id: 0ccabf41-81cd-45e7-bb26-335edaa2ee47 }, name: Name("external-dns-0ccabf41-81cd-45e7-bb26-335edaa2ee47"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 224, 221])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: b845b0e9-661e-4fa1-9292-0c7dce84b88d, kind: Service { id: 38a08dd6-6a75-4558-b4f8-df853f4d03d9 }, name: Name("external-dns-38a08dd6-6a75-4558-b4f8-df853f4d03d9"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 179, 43])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }, ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("80"), L4PortRange("443")]), filter_protocols: Some([Tcp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 559b827c-9574-475a-93d6-1045f64398e6, kind: Service { id: 85a39cf9-9419-458f-af2e-c8b40d8f5f79 }, name: Name("nexus-85a39cf9-9419-458f-af2e-c8b40d8f5f79"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 232, 9])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 715a8bac-a423-4460-aac2-b8c76af6bb34, kind: Service { id: ddd82634-2844-4f97-9746-07099044c19d }, name: Name("nexus-ddd82634-2844-4f97-9746-07099044c19d"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 227, 84])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 76144c16-58bd-4abd-b758-565497d9f6b8, kind: Service { id: 53ef6679-65e8-4486-9401-d6fa4ff2e8e4 }, name: Name("nexus-53ef6679-65e8-4486-9401-d6fa4ff2e8e4"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 174, 62])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }]
112992025-05-08T22:13:38.848ZDEBGnexus (service-firewall-rule-progator): resolved 2 rules for sleds background_task = service_firewall_rule_propagation
113002025-05-08T22:13:38.850ZINFOnexus (BackgroundTasks): instance reincarnation completed successfully background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:113 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0
113012025-05-08T22:13:38.850ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 563.933941ms iteration = 16 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
113022025-05-08T22:13:38.921ZDEBGnexus (service-firewall-rule-progator): resolved sleds for vpc oxide-services background_task = service_firewall_rule_propagation vpc_to_sled = [Sled { identity: SledIdentity { id: 2f2f2c32-8d58-4801-a669-9618eac0c3f7, time_created: 2025-05-08T21:59:12.125305Z, time_modified: 2025-05-08T21:59:12.125305Z }, time_deleted: None, rcgen: Generation(Generation(19)), rack_id: 64f3d6f4-89b8-48aa-a622-b684e261667e, is_scrimlet: true, serial_number: "sock", part_number: "i86pc", revision: SqlU32(0), usable_hardware_threads: SqlU32(48), usable_physical_ram: ByteCount(ByteCount(102952026112)), reservoir_size: ByteCount(ByteCount(56094621696)), ip: fd00:1122:3344:101::1, port: SqlU16(12345), last_used_address: fd00:1122:3344:101::1:0, policy: InService, state: Active, sled_agent_gen: Generation(Generation(1)), repo_depot_port: SqlU16(12348) }]
113032025-05-08T22:13:38.921ZDEBGnexus (service-firewall-rule-progator): sending firewall rules to sled agents background_task = service_firewall_rule_propagation
113042025-05-08T22:13:38.921ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
113052025-05-08T22:13:38.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
113062025-05-08T22:13:38.924ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
113072025-05-08T22:13:38.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
113082025-05-08T22:13:38.925ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
113092025-05-08T22:13:38.930ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Sled { parent: Fleet, key: 2f2f2c32-8d58-4801-a669-9618eac0c3f7, lookup_type: ById(2f2f2c32-8d58-4801-a669-9618eac0c3f7) } result = Ok(())
113102025-05-08T22:13:38.955ZDEBGnexus (BackgroundTasks): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
113112025-05-08T22:13:38.956ZDEBGnexus (BackgroundTasks): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed25a350-4e3d-4323-9ccb-95131aa8ccd1", "content-length": "691", "date": "Thu, 08 May 2025 22:13:38 GMT"} })
113122025-05-08T22:13:38.956ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
113132025-05-08T22:13:38.956ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
113142025-05-08T22:13:38.956ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
113152025-05-08T22:13:38.957ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 670.990907ms iteration = 31 status = {}
113162025-05-08T22:13:38.996ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
113172025-05-08T22:13:38.997ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "bb5140a0-56d6-492e-a385-ff83d3225def", "date": "Thu, 08 May 2025 22:13:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
113182025-05-08T22:13:38.997ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
113192025-05-08T22:13:38.999ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "2ea68b38-4151-45ea-b82f-d5003fc0778d", "date": "Thu, 08 May 2025 22:13:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
113202025-05-08T22:13:38.999ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
113212025-05-08T22:13:39.001ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "0078ba1a-cb94-47e3-9e56-203645aed4f9", "date": "Thu, 08 May 2025 22:13:38 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
113222025-05-08T22:13:39.001ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
113232025-05-08T22:13:39.003ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
113242025-05-08T22:13:39.003ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 732.143861ms iteration = 18 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
113252025-05-08T22:13:39.013ZDEBGnexus (service-firewall-rule-progator): client request SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = service_firewall_rule_propagation body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules
113262025-05-08T22:13:39.046ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
113272025-05-08T22:13:39.047ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
113282025-05-08T22:13:39.049ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: cd0b930c-534b-46ad-ac3b-2eaa94b0bf2c, rack_id: 64f3d6f4-89b8-48aa-a622-b684e261667e, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(65f83a36-18fb-49b7-9914-87c8517d200e) } rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
113292025-05-08T22:13:39.049ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
113302025-05-08T22:13:39.049ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
113312025-05-08T22:13:39.059ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1833 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
113322025-05-08T22:13:39.059ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1860 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 65f83a36-18fb-49b7-9914-87c8517d200e
113332025-05-08T22:13:39.059ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
113342025-05-08T22:13:39.059ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
113352025-05-08T22:13:39.059ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
113362025-05-08T22:13:39.059ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
113372025-05-08T22:13:39.059ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
113382025-05-08T22:13:39.059ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2007 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
113392025-05-08T22:13:39.059ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
113402025-05-08T22:13:39.060ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2043 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
113412025-05-08T22:13:39.060ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e
113422025-05-08T22:13:39.060ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
113432025-05-08T22:13:39.060ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
113442025-05-08T22:13:39.062ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch_location = Switch0
113452025-05-08T22:13:39.062ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
113462025-05-08T22:13:39.101ZDEBGnexus (service-firewall-rule-progator): client response SledAgent = 2f2f2c32-8d58-4801-a669-9618eac0c3f7 background_task = service_firewall_rule_propagation result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules", status: 204, headers: {"x-request-id": "ebf0e52f-20e6-400d-86ec-b8e80333ba64", "date": "Thu, 08 May 2025 22:13:38 GMT"} })
113472025-05-08T22:13:39.103ZINFOnexus (service-firewall-rule-progator): updated firewall rules on 1 sleds background_task = service_firewall_rule_propagation file = nexus/networking/src/firewall_rules.rs:495
113482025-05-08T22:13:39.103ZDEBGnexus (service-firewall-rule-progator): successfully propagated service firewall rules background_task = service_firewall_rule_propagation elapsed = 816.774426ms
113492025-05-08T22:13:39.103ZDEBGnexus (BackgroundTasks): activation complete background_task = service_firewall_rule_propagation elapsed = 816.785726ms iteration = 4 status = {}
113502025-05-08T22:13:39.200ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "3651bc71-18b9-4ed0-b06b-778a0e82ce32", "date": "Thu, 08 May 2025 22:13:38 GMT"} })
113512025-05-08T22:13:39.200ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e switch = Switch0
113522025-05-08T22:13:39.200ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
113532025-05-08T22:13:39.204ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 64f3d6f4-89b8-48aa-a622-b684e261667e result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "ebbf5b2a-926d-49a9-9766-742a382dc69c", "content-length": "682", "date": "Thu, 08 May 2025 22:13:38 GMT"} })
113542025-05-08T22:13:39.204ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
113552025-05-08T22:13:39.204ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
113562025-05-08T22:13:39.208ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
113572025-05-08T22:13:39.208ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
113582025-05-08T22:13:39.211ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
113592025-05-08T22:13:39.212ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
113602025-05-08T22:13:39.214ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
113612025-05-08T22:13:39.214ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
113622025-05-08T22:13:39.221ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 935.116222ms iteration = 31 status = {}
113632025-05-08T22:13:41.941ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 251 local_addr = [fd00:1122:3344:101::b]:50702 method = GET remote_addr = [fd00:1122:3344:101::d]:51203 req_id = 77c7737b-0733-48bd-ba28-4bb679d5c075 response_code = 200 uri = /85a39cf9-9419-458f-af2e-c8b40d8f5f79
113642025-05-08T22:13:48.273ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 92 reason = Timeout
113652025-05-08T22:13:48.273ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
113662025-05-08T22:13:48.273ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
113672025-05-08T22:13:48.274ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
113682025-05-08T22:13:48.277ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
113692025-05-08T22:13:48.277ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
113702025-05-08T22:13:48.277ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
113712025-05-08T22:13:48.279ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
113722025-05-08T22:13:48.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
113732025-05-08T22:13:48.281ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
113742025-05-08T22:13:48.283ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2, lookup_type: ById(d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2) } result = Ok(())
113752025-05-08T22:13:48.283ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
113762025-05-08T22:13:48.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
113772025-05-08T22:13:48.286ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 92 reason = Timeout
113782025-05-08T22:13:48.286ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
113792025-05-08T22:13:48.286ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
113802025-05-08T22:13:48.291ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 original_time_created = 2025-05-08 21:58:32.829969 UTC target_id = d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2 time_created = 2025-05-08 21:58:32.829969 UTC
113812025-05-08T22:13:48.291ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 18.804751ms iteration = 92 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"d6ca719b-a2d1-4e42-a13d-b1f38f25e0b2","time_created":"2025-05-08T21:58:32.829969Z"}
113822025-05-08T22:13:48.318ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
113832025-05-08T22:13:48.318ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
113842025-05-08T22:13:48.319ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 33.404627ms iteration = 92 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}