Max level shown:
12025-01-10T05:15:02.657ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 56 reason = Timeout
22025-01-10T05:15:02.657ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 56 reason = Timeout
32025-01-10T05:15:02.657ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
42025-01-10T05:15:02.657ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
52025-01-10T05:15:02.658ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
62025-01-10T05:15:02.658ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
72025-01-10T05:15:02.660ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
82025-01-10T05:15:02.660ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
92025-01-10T05:15:02.661ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102025-01-10T05:15:02.662ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 5.176588ms iteration = 56 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
112025-01-10T05:15:02.664ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
122025-01-10T05:15:02.664ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
132025-01-10T05:15:02.664ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
142025-01-10T05:15:02.667ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
152025-01-10T05:15:02.667ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
162025-01-10T05:15:02.669ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
172025-01-10T05:15:02.673ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 43ea24da-5730-4a85-9a8e-76459f9b69b1, lookup_type: ById(43ea24da-5730-4a85-9a8e-76459f9b69b1) } result = Ok(())
182025-01-10T05:15:02.673ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
192025-01-10T05:15:02.673ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
202025-01-10T05:15:02.692ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 original_time_created = 2025-01-10 05:05:43.319091 UTC target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 time_created = 2025-01-10 05:05:43.319091 UTC
212025-01-10T05:15:02.692ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 34.595043ms iteration = 56 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"43ea24da-5730-4a85-9a8e-76459f9b69b1","time_created":"2025-01-10T05:05:43.319091Z"}
222025-01-10T05:15:07.550ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 332 local_addr = [fd00:1122:3344:101::c]:57161 method = GET remote_addr = [fd00:1122:3344:101::d]:46894 req_id = b54b4edf-5c6a-4e96-8d4b-022ec36a8cdf response_code = 200 uri = /3ca89dd1-7ada-4fe6-8600-a84d41dfa9d8
232025-01-10T05:15:12.657ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 57 reason = Timeout
242025-01-10T05:15:12.657ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 57 reason = Timeout
252025-01-10T05:15:12.657ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
262025-01-10T05:15:12.657ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
272025-01-10T05:15:12.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
282025-01-10T05:15:12.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
292025-01-10T05:15:12.659ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
302025-01-10T05:15:12.659ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
312025-01-10T05:15:12.660ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
322025-01-10T05:15:12.661ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.753984ms iteration = 57 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
332025-01-10T05:15:12.662ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
342025-01-10T05:15:12.662ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
352025-01-10T05:15:12.662ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
362025-01-10T05:15:12.663ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
372025-01-10T05:15:12.663ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
382025-01-10T05:15:12.665ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
392025-01-10T05:15:12.668ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 43ea24da-5730-4a85-9a8e-76459f9b69b1, lookup_type: ById(43ea24da-5730-4a85-9a8e-76459f9b69b1) } result = Ok(())
402025-01-10T05:15:12.668ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
412025-01-10T05:15:12.668ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
422025-01-10T05:15:12.678ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 original_time_created = 2025-01-10 05:05:43.319091 UTC target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 time_created = 2025-01-10 05:05:43.319091 UTC
432025-01-10T05:15:12.678ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.536489ms iteration = 57 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"43ea24da-5730-4a85-9a8e-76459f9b69b1","time_created":"2025-01-10T05:05:43.319091Z"}
442025-01-10T05:15:17.549ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 300 local_addr = [fd00:1122:3344:101::c]:57161 method = GET remote_addr = [fd00:1122:3344:101::d]:46894 req_id = 94b3a7c0-7daf-414c-8957-45061b457e75 response_code = 200 uri = /3ca89dd1-7ada-4fe6-8600-a84d41dfa9d8
452025-01-10T05:15:22.657ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 21 reason = Timeout
462025-01-10T05:15:22.657ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 20 reason = Timeout
472025-01-10T05:15:22.657ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
482025-01-10T05:15:22.657ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 20 reason = Timeout
492025-01-10T05:15:22.657ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
502025-01-10T05:15:22.657ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
512025-01-10T05:15:22.657ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 58 reason = Timeout
522025-01-10T05:15:22.658ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
532025-01-10T05:15:22.658ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 20 reason = Timeout
542025-01-10T05:15:22.658ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
552025-01-10T05:15:22.658ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 20 reason = Timeout
562025-01-10T05:15:22.658ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
572025-01-10T05:15:22.658ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 20 reason = Timeout
582025-01-10T05:15:22.658ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
592025-01-10T05:15:22.658ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
602025-01-10T05:15:22.658ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 20 reason = Timeout
612025-01-10T05:15:22.658ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 20 reason = Timeout
622025-01-10T05:15:22.658ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
632025-01-10T05:15:22.658ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
642025-01-10T05:15:22.658ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 20 reason = Timeout
652025-01-10T05:15:22.658ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
662025-01-10T05:15:22.658ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
672025-01-10T05:15:22.658ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 20 reason = Timeout
682025-01-10T05:15:22.658ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
692025-01-10T05:15:22.658ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 20 reason = Timeout
702025-01-10T05:15:22.658ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
712025-01-10T05:15:22.658ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 20 reason = Timeout
722025-01-10T05:15:22.658ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 58 reason = Timeout
732025-01-10T05:15:22.658ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 20 reason = Timeout
742025-01-10T05:15:22.658ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 20 reason = Timeout
752025-01-10T05:15:22.658ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
762025-01-10T05:15:22.658ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
772025-01-10T05:15:22.658ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
782025-01-10T05:15:22.658ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
792025-01-10T05:15:22.658ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
802025-01-10T05:15:22.658ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
812025-01-10T05:15:22.658ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
822025-01-10T05:15:22.658ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
832025-01-10T05:15:22.658ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
842025-01-10T05:15:22.658ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
852025-01-10T05:15:22.658ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
862025-01-10T05:15:22.658ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
872025-01-10T05:15:22.658ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
882025-01-10T05:15:22.658ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
892025-01-10T05:15:22.658ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
902025-01-10T05:15:22.658ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
912025-01-10T05:15:22.658ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
922025-01-10T05:15:22.658ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
932025-01-10T05:15:22.659ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 20 reason = Timeout
942025-01-10T05:15:22.659ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
952025-01-10T05:15:22.659ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
962025-01-10T05:15:22.660ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
972025-01-10T05:15:22.661ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
982025-01-10T05:15:22.661ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
992025-01-10T05:15:22.661ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.821694ms iteration = 20 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
1002025-01-10T05:15:22.662ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 26933, tv_nsec: 168243524 } })
1012025-01-10T05:15:22.662ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1022025-01-10T05:15:22.662ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1103
1032025-01-10T05:15:22.663ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
1042025-01-10T05:15:22.663ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
1052025-01-10T05:15:22.663ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
1062025-01-10T05:15:22.664ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
1072025-01-10T05:15:22.664ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
1082025-01-10T05:15:22.664ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
1092025-01-10T05:15:22.664ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
1102025-01-10T05:15:22.664ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
1112025-01-10T05:15:22.665ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
1122025-01-10T05:15:22.665ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
1132025-01-10T05:15:22.665ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
1142025-01-10T05:15:22.665ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
1152025-01-10T05:15:22.666ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1162025-01-10T05:15:22.667ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1172025-01-10T05:15:22.667ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
1182025-01-10T05:15:22.668ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
1192025-01-10T05:15:22.668ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
1202025-01-10T05:15:22.668ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
1212025-01-10T05:15:22.668ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
1222025-01-10T05:15:22.668ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
1232025-01-10T05:15:22.668ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
1242025-01-10T05:15:22.668ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
1252025-01-10T05:15:22.670ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
1262025-01-10T05:15:22.670ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1272025-01-10T05:15:22.670ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
1282025-01-10T05:15:22.670ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
1292025-01-10T05:15:22.670ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1302025-01-10T05:15:22.670ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
1312025-01-10T05:15:22.675ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 16.381191ms iteration = 20 status = {"errors":[],"finish_invoked_ok":[]}
1322025-01-10T05:15:22.675ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
1332025-01-10T05:15:22.675ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
1342025-01-10T05:15:22.675ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
1352025-01-10T05:15:22.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
1362025-01-10T05:15:22.676ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 26933, tv_nsec: 182653942 } })
1372025-01-10T05:15:22.676ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
1382025-01-10T05:15:22.677ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
1392025-01-10T05:15:22.677ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1103
1402025-01-10T05:15:22.678ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 20.487055ms iteration = 58 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
1412025-01-10T05:15:22.678ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 26933, tv_nsec: 184445920 } })
1422025-01-10T05:15:22.678ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1103 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08
1432025-01-10T05:15:22.680ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1442025-01-10T05:15:22.680ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1452025-01-10T05:15:22.680ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
1462025-01-10T05:15:22.680ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
1472025-01-10T05:15:22.682ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1482025-01-10T05:15:22.690ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 43ea24da-5730-4a85-9a8e-76459f9b69b1, lookup_type: ById(43ea24da-5730-4a85-9a8e-76459f9b69b1) } result = Ok(())
1492025-01-10T05:15:22.690ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
1502025-01-10T05:15:22.690ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
1512025-01-10T05:15:22.691ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
1522025-01-10T05:15:22.692ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
1532025-01-10T05:15:22.694ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1542025-01-10T05:15:22.696ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
1552025-01-10T05:15:22.696ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
1562025-01-10T05:15:22.696ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
1572025-01-10T05:15:22.697ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1582025-01-10T05:15:22.699ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
1592025-01-10T05:15:22.699ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
1602025-01-10T05:15:22.700ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
1612025-01-10T05:15:22.703ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 original_time_created = 2025-01-10 05:05:43.319091 UTC target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 time_created = 2025-01-10 05:05:43.319091 UTC
1622025-01-10T05:15:22.703ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 45.021481ms iteration = 58 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"43ea24da-5730-4a85-9a8e-76459f9b69b1","time_created":"2025-01-10T05:05:43.319091Z"}
1632025-01-10T05:15:22.705ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 47.402208ms iteration = 20 status = {"errors":[],"garbage_collect_requested":[]}
1642025-01-10T05:15:22.708ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1652025-01-10T05:15:22.708ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1662025-01-10T05:15:22.710ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1672025-01-10T05:15:22.711ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1682025-01-10T05:15:22.711ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1692025-01-10T05:15:22.713ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
1702025-01-10T05:15:22.713ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
1712025-01-10T05:15:22.713ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
1722025-01-10T05:15:22.714ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
1732025-01-10T05:15:22.714ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
1742025-01-10T05:15:22.714ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 56.739271ms iteration = 20 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
1752025-01-10T05:15:22.720ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
1762025-01-10T05:15:22.720ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
1772025-01-10T05:15:22.720ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
1782025-01-10T05:15:22.720ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
1792025-01-10T05:15:22.720ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
1802025-01-10T05:15:22.720ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 62.700135ms iteration = 20 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
1812025-01-10T05:15:22.723ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1822025-01-10T05:15:22.725ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
1832025-01-10T05:15:22.725ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
1842025-01-10T05:15:22.725ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
1852025-01-10T05:15:22.726ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
1862025-01-10T05:15:22.727ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
1872025-01-10T05:15:22.795ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1111 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 zone_address = fd00:1122:3344:101::2
1882025-01-10T05:15:22.795ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
1892025-01-10T05:15:22.795ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "18d19576-1774-400b-bc71-116637342c35", "content-length": "26", "date": "Fri, 10 Jan 2025 05:15:22 GMT"} })
1902025-01-10T05:15:22.796ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1116 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
1912025-01-10T05:15:22.798ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1148 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 79fde928-482f-41ab-aee3-3f2478145f08
1922025-01-10T05:15:22.811ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1111 zone_address = fd00:1122:3344:101::2
1932025-01-10T05:15:22.811ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
1942025-01-10T05:15:22.811ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "87a9d230-1024-4887-8eb0-d79c2f6c4366", "content-length": "26", "date": "Fri, 10 Jan 2025 05:15:22 GMT"} })
1952025-01-10T05:15:22.811ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1116 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
1962025-01-10T05:15:22.813ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1148 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
1972025-01-10T05:15:22.822ZDEBGnexus (BackgroundTasks): client request SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
1982025-01-10T05:15:22.823ZDEBGnexus (BackgroundTasks): client response SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9772c7b-58cf-4cac-967a-7a08a681852f", "content-length": "691", "date": "Fri, 10 Jan 2025 05:15:22 GMT"} })
1992025-01-10T05:15:22.823ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
2002025-01-10T05:15:22.823ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
2012025-01-10T05:15:22.823ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
2022025-01-10T05:15:22.823ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2032025-01-10T05:15:22.824ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 166.640518ms iteration = 20 status = {}
2042025-01-10T05:15:22.825ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 167.256948ms iteration = 20 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
2052025-01-10T05:15:22.827ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59, lookup_type: ById(a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59) } result = Ok(())
2062025-01-10T05:15:22.827ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
2072025-01-10T05:15:22.827ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
2082025-01-10T05:15:22.829ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 171.291409ms iteration = 20 status = 0
2092025-01-10T05:15:22.868ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1111 zone_address = fd00:1122:3344:101::2
2102025-01-10T05:15:22.868ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
2112025-01-10T05:15:22.869ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "4675046a-3e3b-4951-a5c7-f47c7d869b5e", "content-length": "26", "date": "Fri, 10 Jan 2025 05:15:22 GMT"} })
2122025-01-10T05:15:22.869ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1116 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
2132025-01-10T05:15:22.870ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1148 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
2142025-01-10T05:15:22.876ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
2152025-01-10T05:15:22.876ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
2162025-01-10T05:15:22.876ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 218.403208ms iteration = 20 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
2172025-01-10T05:15:22.876ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 218.360687ms iteration = 20 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
2182025-01-10T05:15:22.930ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
2192025-01-10T05:15:22.930ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "c10c6890-7f37-40ef-9a81-de9facbd47e9", "content-length": "1", "date": "Fri, 10 Jan 2025 05:15:22 GMT"} })
2202025-01-10T05:15:22.932ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
2212025-01-10T05:15:22.932ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
2222025-01-10T05:15:22.934ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 276.610846ms iteration = 20 status = null
2232025-01-10T05:15:22.947ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 289.979751ms iteration = 21 status = {"physical_disks_added":0}
2242025-01-10T05:15:22.950ZINFOnexus (BackgroundTasks): VPC route manager sled a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
2252025-01-10T05:15:22.950ZDEBGnexus (BackgroundTasks): client request SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
2262025-01-10T05:15:22.950ZDEBGnexus (BackgroundTasks): client response SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "a76b4d98-f790-4f01-ac9b-16d7eb1f211f", "content-length": "372", "date": "Fri, 10 Jan 2025 05:15:22 GMT"} })
2272025-01-10T05:15:22.950ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
2282025-01-10T05:15:22.951ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
2292025-01-10T05:15:22.966ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 308.73374ms iteration = 20 status = {}
2302025-01-10T05:15:23.000ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
2312025-01-10T05:15:23.000ZDEBGnexus (BackgroundTasks): client request SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
2322025-01-10T05:15:23.000ZDEBGnexus (BackgroundTasks): client response SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "9d880618-85c9-43c5-8194-efd1672cf9c6", "date": "Fri, 10 Jan 2025 05:15:22 GMT"} })
2332025-01-10T05:15:23.000ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
2342025-01-10T05:15:23.001ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
2352025-01-10T05:15:23.003ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
2362025-01-10T05:15:23.004ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
2372025-01-10T05:15:23.007ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
2382025-01-10T05:15:23.007ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
2392025-01-10T05:15:23.011ZINFOnexus (BackgroundTasks): VPC route manager sled a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
2402025-01-10T05:15:23.011ZDEBGnexus (BackgroundTasks): client request SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
2412025-01-10T05:15:23.011ZDEBGnexus (BackgroundTasks): client response SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "2aa9d070-31b6-4dc3-b84f-c1c0e1be53ad", "date": "Fri, 10 Jan 2025 05:15:22 GMT"} })
2422025-01-10T05:15:23.013ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 355.734744ms iteration = 20 status = {}
2432025-01-10T05:15:23.023ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
2442025-01-10T05:15:23.023ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
2452025-01-10T05:15:23.024ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 5263359f-a80d-45e0-9595-0b15b904e78f, rack_id: 79fde928-482f-41ab-aee3-3f2478145f08, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(77858eb2-1069-4750-be5e-f3dafe20063a) } rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 switch_location = Switch0
2462025-01-10T05:15:23.024ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
2472025-01-10T05:15:23.025ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
2482025-01-10T05:15:23.035ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
2492025-01-10T05:15:23.036ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1a13f84-e9da-4626-9618-efb015fa0bf0", "content-length": "134", "date": "Fri, 10 Jan 2025 05:15:22 GMT"} })
2502025-01-10T05:15:23.036ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
2512025-01-10T05:15:23.036ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 77858eb2-1069-4750-be5e-f3dafe20063a
2522025-01-10T05:15:23.036ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
2532025-01-10T05:15:23.036ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
2542025-01-10T05:15:23.036ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
2552025-01-10T05:15:23.036ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
2562025-01-10T05:15:23.036ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
2572025-01-10T05:15:23.036ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
2582025-01-10T05:15:23.036ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
2592025-01-10T05:15:23.036ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
2602025-01-10T05:15:23.037ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08
2612025-01-10T05:15:23.037ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
2622025-01-10T05:15:23.037ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
2632025-01-10T05:15:23.038ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
2642025-01-10T05:15:23.038ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "2489c542-628b-445c-bf20-2b68d67af464", "content-length": "2", "date": "Fri, 10 Jan 2025 05:15:22 GMT"} })
2652025-01-10T05:15:23.038ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
2662025-01-10T05:15:23.038ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "81a1bcf1-fafd-4b2d-9c97-942fe520d11c", "content-length": "52", "date": "Fri, 10 Jan 2025 05:15:22 GMT"} })
2672025-01-10T05:15:23.038ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 switch_location = Switch0
2682025-01-10T05:15:23.038ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
2692025-01-10T05:15:23.139ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "d86236a1-7df6-4ef4-a03a-9401cbc863dc", "date": "Fri, 10 Jan 2025 05:15:22 GMT"} })
2702025-01-10T05:15:23.139ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 switch = Switch0
2712025-01-10T05:15:23.139ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
2722025-01-10T05:15:23.139ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3d0b104-72c2-4883-93a9-ad9f893ca9c7", "content-length": "682", "date": "Fri, 10 Jan 2025 05:15:23 GMT"} })
2732025-01-10T05:15:23.140ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
2742025-01-10T05:15:23.140ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
2752025-01-10T05:15:23.143ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
2762025-01-10T05:15:23.143ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
2772025-01-10T05:15:23.147ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
2782025-01-10T05:15:23.147ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
2792025-01-10T05:15:23.149ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
2802025-01-10T05:15:23.150ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
2812025-01-10T05:15:23.157ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 499.685597ms iteration = 20 status = {}
2822025-01-10T05:15:27.549ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 271 local_addr = [fd00:1122:3344:101::c]:57161 method = GET remote_addr = [fd00:1122:3344:101::d]:46894 req_id = 95eceec7-bfb2-49eb-8db0-13c84c2c48bb response_code = 200 uri = /3ca89dd1-7ada-4fe6-8600-a84d41dfa9d8
2832025-01-10T05:15:32.657ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 59 reason = Timeout
2842025-01-10T05:15:32.657ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
2852025-01-10T05:15:32.657ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 59 reason = Timeout
2862025-01-10T05:15:32.657ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2872025-01-10T05:15:32.658ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
2882025-01-10T05:15:32.658ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2892025-01-10T05:15:32.660ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2902025-01-10T05:15:32.660ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
2912025-01-10T05:15:32.660ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
2922025-01-10T05:15:32.662ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
2932025-01-10T05:15:32.662ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2942025-01-10T05:15:32.662ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2952025-01-10T05:15:32.664ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2962025-01-10T05:15:32.664ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2972025-01-10T05:15:32.666ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2982025-01-10T05:15:32.668ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 43ea24da-5730-4a85-9a8e-76459f9b69b1, lookup_type: ById(43ea24da-5730-4a85-9a8e-76459f9b69b1) } result = Ok(())
2992025-01-10T05:15:32.668ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3002025-01-10T05:15:32.669ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3012025-01-10T05:15:32.677ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 original_time_created = 2025-01-10 05:05:43.319091 UTC target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 time_created = 2025-01-10 05:05:43.319091 UTC
3022025-01-10T05:15:32.677ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 19.887165ms iteration = 59 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"43ea24da-5730-4a85-9a8e-76459f9b69b1","time_created":"2025-01-10T05:05:43.319091Z"}
3032025-01-10T05:15:32.689ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 31.845503ms iteration = 59 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
3042025-01-10T05:15:37.549ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 261 local_addr = [fd00:1122:3344:101::c]:57161 method = GET remote_addr = [fd00:1122:3344:101::d]:46894 req_id = c66fba20-28b3-423c-bb5e-8fc800fd601c response_code = 200 uri = /3ca89dd1-7ada-4fe6-8600-a84d41dfa9d8
3052025-01-10T05:15:42.657ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 60 reason = Timeout
3062025-01-10T05:15:42.657ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 60 reason = Timeout
3072025-01-10T05:15:42.657ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3082025-01-10T05:15:42.657ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
3092025-01-10T05:15:42.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3102025-01-10T05:15:42.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
3112025-01-10T05:15:42.659ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3122025-01-10T05:15:42.659ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
3132025-01-10T05:15:42.659ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
3142025-01-10T05:15:42.661ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.405053ms iteration = 60 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
3152025-01-10T05:15:42.662ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
3162025-01-10T05:15:42.662ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3172025-01-10T05:15:42.662ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3182025-01-10T05:15:42.664ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3192025-01-10T05:15:42.664ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3202025-01-10T05:15:42.665ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3212025-01-10T05:15:42.668ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 43ea24da-5730-4a85-9a8e-76459f9b69b1, lookup_type: ById(43ea24da-5730-4a85-9a8e-76459f9b69b1) } result = Ok(())
3222025-01-10T05:15:42.668ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3232025-01-10T05:15:42.668ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3242025-01-10T05:15:42.681ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 original_time_created = 2025-01-10 05:05:43.319091 UTC target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 time_created = 2025-01-10 05:05:43.319091 UTC
3252025-01-10T05:15:42.681ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 24.02849ms iteration = 60 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"43ea24da-5730-4a85-9a8e-76459f9b69b1","time_created":"2025-01-10T05:05:43.319091Z"}
3262025-01-10T05:15:47.549ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 288 local_addr = [fd00:1122:3344:101::c]:57161 method = GET remote_addr = [fd00:1122:3344:101::d]:46894 req_id = adcb8a23-3165-4c37-a664-d77e2325a204 response_code = 200 uri = /3ca89dd1-7ada-4fe6-8600-a84d41dfa9d8
3272025-01-10T05:15:52.656ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 61 reason = Timeout
3282025-01-10T05:15:52.656ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
3292025-01-10T05:15:52.656ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 11 reason = Timeout
3302025-01-10T05:15:52.656ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-01-10 05:05:52.656620070 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
3312025-01-10T05:15:52.656ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 11 reason = Timeout
3322025-01-10T05:15:52.656ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
3332025-01-10T05:15:52.656ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
3342025-01-10T05:15:52.656ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
3352025-01-10T05:15:52.656ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 11 reason = Timeout
3362025-01-10T05:15:52.656ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
3372025-01-10T05:15:52.656ZDEBGnexus (BackgroundTasks): activating background_task = saga_recovery iteration = 2 reason = Timeout
3382025-01-10T05:15:52.656ZDEBGnexus (SagaRecoverer): listing candidate sagas for recovery
3392025-01-10T05:15:52.656ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 11 reason = Timeout
3402025-01-10T05:15:52.656ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
3412025-01-10T05:15:52.656ZDEBGnexus (SagaRecoverer): roles roles = RoleSet { roles: {} }
3422025-01-10T05:15:52.656ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 21 reason = Timeout
3432025-01-10T05:15:52.656ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
3442025-01-10T05:15:52.656ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
3452025-01-10T05:15:52.656ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 21 reason = Timeout
3462025-01-10T05:15:52.656ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
3472025-01-10T05:15:52.656ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 21 reason = Timeout
3482025-01-10T05:15:52.656ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
3492025-01-10T05:15:52.656ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 11 reason = Timeout
3502025-01-10T05:15:52.656ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 11 reason = Timeout
3512025-01-10T05:15:52.656ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 22 reason = Timeout
3522025-01-10T05:15:52.656ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 21 reason = Timeout
3532025-01-10T05:15:52.656ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
3542025-01-10T05:15:52.656ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 21 reason = Timeout
3552025-01-10T05:15:52.656ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
3562025-01-10T05:15:52.656ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 21 reason = Timeout
3572025-01-10T05:15:52.656ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 11 reason = Timeout
3582025-01-10T05:15:52.656ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
3592025-01-10T05:15:52.656ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
3602025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 11 reason = Timeout
3612025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 11 reason = Timeout
3622025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
3632025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
3642025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
3652025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 12 reason = Timeout
3662025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 13 reason = Timeout
3672025-01-10T05:15:52.657ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:83 reason = blueprint disabled target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1
3682025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 13.921\u{b5}s iteration = 12 status = {"enabled":false,"target_id":"43ea24da-5730-4a85-9a8e-76459f9b69b1"}
3692025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 61 reason = Timeout
3702025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 21 reason = Timeout
3712025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 21 reason = Timeout
3722025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 21 reason = Timeout
3732025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
3742025-01-10T05:15:52.657ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
3752025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
3762025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
3772025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3782025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
3792025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 13 reason = Timeout
3802025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): activating background_task = inventory_collection iteration = 2 reason = Timeout
3812025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
3822025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 21 reason = Timeout
3832025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
3842025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 21 reason = Timeout
3852025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
3862025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 11 reason = Timeout
3872025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
3882025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 21 reason = Timeout
3892025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
3902025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 21 reason = Timeout
3912025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
3922025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): activating background_task = service_firewall_rule_propagation iteration = 3 reason = Timeout
3932025-01-10T05:15:52.657ZDEBGnexus (service-firewall-rule-progator): starting background task for service firewall rule propagation background_task = service_firewall_rule_propagation
3942025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
3952025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
3962025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
3972025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
3982025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
3992025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
4002025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
4012025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
4022025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
4032025-01-10T05:15:52.657ZDEBGnexus (SagaRecoverer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000003a8a, .. }) resource = Database result = Ok(())
4042025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
4052025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
4062025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
4072025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
4082025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
4092025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
4102025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
4112025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
4122025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
4132025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
4142025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4152025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
4162025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
4172025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
4182025-01-10T05:15:52.657ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("53514ca5-28a5-4b76-afd8-362873acf1d4.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("b3737e73-ceea-4b83-966e-05b55558955a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("e93887bd-ffe8-46b8-ad6f-92e0665448ab.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("e93887bd-ffe8-46b8-ad6f-92e0665448ab.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 26963, tv_nsec: 163832961 } })
4192025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
4202025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
4212025-01-10T05:15:52.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
4222025-01-10T05:15:52.657ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("30a043aa-79b8-40f0-889b-b1aa4686508a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("95b462f2-3c97-4463-9c08-5404518f33d2.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("95b462f2-3c97-4463-9c08-5404518f33d2.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 26963, tv_nsec: 163869842 } })
4232025-01-10T05:15:52.658ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
4242025-01-10T05:15:52.658ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.369085ms iteration = 11 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
4252025-01-10T05:15:52.658ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
4262025-01-10T05:15:52.658ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.439877ms iteration = 11 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353"]}
4272025-01-10T05:15:52.658ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
4282025-01-10T05:15:52.658ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
4292025-01-10T05:15:52.659ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 21 reason = Timeout
4302025-01-10T05:15:52.659ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
4312025-01-10T05:15:52.659ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
4322025-01-10T05:15:52.659ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-01-10 05:05:52.656620070 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
4332025-01-10T05:15:52.659ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 2.986427ms iteration = 11 status = {"expiration":"2025-01-10T05:05:52.656620070Z","pruned":[]}
4342025-01-10T05:15:52.660ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
4352025-01-10T05:15:52.660ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
4362025-01-10T05:15:52.660ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4372025-01-10T05:15:52.661ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.476495ms iteration = 21 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
4382025-01-10T05:15:52.661ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 4.646511ms iteration = 11 status = {"errors":[],"found_port_ok":[]}
4392025-01-10T05:15:52.662ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4402025-01-10T05:15:52.663ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 26963, tv_nsec: 169363191 } })
4412025-01-10T05:15:52.663ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1103
4422025-01-10T05:15:52.663ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
4432025-01-10T05:15:52.663ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
4442025-01-10T05:15:52.663ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
4452025-01-10T05:15:52.664ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
4462025-01-10T05:15:52.664ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
4472025-01-10T05:15:52.664ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
4482025-01-10T05:15:52.666ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
4492025-01-10T05:15:52.667ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 26963, tv_nsec: 173233096 } })
4502025-01-10T05:15:52.667ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
4512025-01-10T05:15:52.667ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1103
4522025-01-10T05:15:52.668ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
4532025-01-10T05:15:52.668ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:171 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
4542025-01-10T05:15:52.668ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 11.379809ms iteration = 11 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
4552025-01-10T05:15:52.671ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
4562025-01-10T05:15:52.671ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
4572025-01-10T05:15:52.674ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 17.701975ms iteration = 61 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
4582025-01-10T05:15:52.676ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
4592025-01-10T05:15:52.677ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
4602025-01-10T05:15:52.702ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4612025-01-10T05:15:52.705ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
4622025-01-10T05:15:52.705ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
4632025-01-10T05:15:52.705ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
4642025-01-10T05:15:52.706ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
4652025-01-10T05:15:52.706ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
4662025-01-10T05:15:52.706ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
4672025-01-10T05:15:52.706ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
4682025-01-10T05:15:52.706ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4692025-01-10T05:15:52.708ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4702025-01-10T05:15:52.708ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4712025-01-10T05:15:52.709ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
4722025-01-10T05:15:52.709ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
4732025-01-10T05:15:52.710ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
4742025-01-10T05:15:52.710ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
4752025-01-10T05:15:52.710ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4762025-01-10T05:15:52.710ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
4772025-01-10T05:15:52.710ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 53.803496ms iteration = 21 status = {"errors":[],"garbage_collect_requested":[]}
4782025-01-10T05:15:52.710ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4792025-01-10T05:15:52.711ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
4802025-01-10T05:15:52.711ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
4812025-01-10T05:15:52.711ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
4822025-01-10T05:15:52.712ZINFOnexus (SagaRecoverer): listed in-progress sagas count = 0 file = nexus/src/app/background/tasks/saga_recovery.rs:438
4832025-01-10T05:15:52.712ZDEBGnexus (BackgroundTasks): activation complete background_task = saga_recovery elapsed = 55.4735ms iteration = 2 status = {"last_pass":{"Success":{"nfailed":0,"nfound":0,"nrecovered":0,"nremoved":0,"nskipped":0}},"ntotal_failures":0,"ntotal_finished":0,"ntotal_recovered":0,"ntotal_sec_errors_bad_state":0,"ntotal_sec_errors_missing":0,"ntotal_started":0,"recent_failures":[],"recent_recoveries":[]}
4842025-01-10T05:15:52.712ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
4852025-01-10T05:15:52.712ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4862025-01-10T05:15:52.712ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
4872025-01-10T05:15:52.712ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
4882025-01-10T05:15:52.712ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
4892025-01-10T05:15:52.713ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
4902025-01-10T05:15:52.713ZDEBGnexus (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
4912025-01-10T05:15:52.713ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
4922025-01-10T05:15:52.714ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
4932025-01-10T05:15:52.714ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
4942025-01-10T05:15:52.714ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4952025-01-10T05:15:52.714ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4962025-01-10T05:15:52.715ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
4972025-01-10T05:15:52.715ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
4982025-01-10T05:15:52.715ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
4992025-01-10T05:15:52.717ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5002025-01-10T05:15:52.718ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5012025-01-10T05:15:52.721ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
5022025-01-10T05:15:52.721ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
5032025-01-10T05:15:52.721ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
5042025-01-10T05:15:52.721ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
5052025-01-10T05:15:52.721ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
5062025-01-10T05:15:52.722ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
5072025-01-10T05:15:52.722ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 62.953323ms iteration = 21 status = {"errors":[],"finish_invoked_ok":[]}
5082025-01-10T05:15:52.722ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 65.439324ms iteration = 21 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
5092025-01-10T05:15:52.724ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
5102025-01-10T05:15:52.724ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
5112025-01-10T05:15:52.727ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
5122025-01-10T05:15:52.728ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5132025-01-10T05:15:52.728ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
5142025-01-10T05:15:52.728ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
5152025-01-10T05:15:52.728ZDEBGnexus (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(9d7c4396-d396-4b89-99d6-0949b44fccb7, 0), (71fdc11f-4757-4abd-8ec7-cf8c4ffed5d3, 0), (9a21c758-c0b2-468f-bef5-889eae3f169d, 0)]
5162025-01-10T05:15:52.728ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
5172025-01-10T05:15:52.728ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
5182025-01-10T05:15:52.728ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12225, target: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 26963, tv_nsec: 234991050 } })
5192025-01-10T05:15:52.728ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
5202025-01-10T05:15:52.728ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
5212025-01-10T05:15:52.729ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
5222025-01-10T05:15:52.731ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
5232025-01-10T05:15:52.733ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
5242025-01-10T05:15:52.733ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
5252025-01-10T05:15:52.733ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 26963, tv_nsec: 239961342 } })
5262025-01-10T05:15:52.733ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5272025-01-10T05:15:52.734ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1103 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08
5282025-01-10T05:15:52.734ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5292025-01-10T05:15:52.734ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5302025-01-10T05:15:52.738ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
5312025-01-10T05:15:52.738ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
5322025-01-10T05:15:52.738ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
5332025-01-10T05:15:52.738ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
5342025-01-10T05:15:52.738ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
5352025-01-10T05:15:52.738ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
5362025-01-10T05:15:52.740ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 43ea24da-5730-4a85-9a8e-76459f9b69b1, lookup_type: ById(43ea24da-5730-4a85-9a8e-76459f9b69b1) } result = Ok(())
5372025-01-10T05:15:52.740ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
5382025-01-10T05:15:52.740ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
5392025-01-10T05:15:52.740ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
5402025-01-10T05:15:52.740ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
5412025-01-10T05:15:52.742ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5422025-01-10T05:15:52.747ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
5432025-01-10T05:15:52.747ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
5442025-01-10T05:15:52.747ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
5452025-01-10T05:15:52.749ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
5462025-01-10T05:15:52.749ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
5472025-01-10T05:15:52.751ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
5482025-01-10T05:15:52.751ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
5492025-01-10T05:15:52.752ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5502025-01-10T05:15:52.753ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
5512025-01-10T05:15:52.754ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
5522025-01-10T05:15:52.754ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5532025-01-10T05:15:52.754ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5542025-01-10T05:15:52.757ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
5552025-01-10T05:15:52.757ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
5562025-01-10T05:15:52.757ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 101.245576ms iteration = 11 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
5572025-01-10T05:15:52.758ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
5582025-01-10T05:15:52.758ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
5592025-01-10T05:15:52.759ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
5602025-01-10T05:15:52.760ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 original_time_created = 2025-01-10 05:05:43.319091 UTC target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 time_created = 2025-01-10 05:05:43.319091 UTC
5612025-01-10T05:15:52.760ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 104.064367ms iteration = 61 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"43ea24da-5730-4a85-9a8e-76459f9b69b1","time_created":"2025-01-10T05:05:43.319091Z"}
5622025-01-10T05:15:52.763ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5632025-01-10T05:15:52.763ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5642025-01-10T05:15:52.764ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
5652025-01-10T05:15:52.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
5662025-01-10T05:15:52.766ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
5672025-01-10T05:15:52.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
5682025-01-10T05:15:52.766ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
5692025-01-10T05:15:52.766ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
5702025-01-10T05:15:52.766ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
5712025-01-10T05:15:52.767ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
5722025-01-10T05:15:52.767ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
5732025-01-10T05:15:52.767ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
5742025-01-10T05:15:52.767ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
5752025-01-10T05:15:52.767ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
5762025-01-10T05:15:52.767ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
5772025-01-10T05:15:52.768ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
5782025-01-10T05:15:52.768ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
5792025-01-10T05:15:52.768ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5802025-01-10T05:15:52.769ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
5812025-01-10T05:15:52.769ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
5822025-01-10T05:15:52.769ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 112.310815ms iteration = 21 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
5832025-01-10T05:15:52.770ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
5842025-01-10T05:15:52.770ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
5852025-01-10T05:15:52.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
5862025-01-10T05:15:52.770ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
5872025-01-10T05:15:52.770ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
5882025-01-10T05:15:52.770ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
5892025-01-10T05:15:52.770ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
5902025-01-10T05:15:52.771ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
5912025-01-10T05:15:52.771ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
5922025-01-10T05:15:52.772ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
5932025-01-10T05:15:52.772ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
5942025-01-10T05:15:52.773ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
5952025-01-10T05:15:52.773ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
5962025-01-10T05:15:52.774ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
5972025-01-10T05:15:52.775ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5982025-01-10T05:15:52.775ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 118.784315ms iteration = 21 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
5992025-01-10T05:15:52.776ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
6002025-01-10T05:15:52.776ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
6012025-01-10T05:15:52.777ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
6022025-01-10T05:15:52.777ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 70f09b61-dde8-4da2-af2f-06138d5a975e dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
6032025-01-10T05:15:52.777ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
6042025-01-10T05:15:52.778ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
6052025-01-10T05:15:52.778ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6062025-01-10T05:15:52.778ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
6072025-01-10T05:15:52.778ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
6082025-01-10T05:15:52.778ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
6092025-01-10T05:15:52.778ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
6102025-01-10T05:15:52.778ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 121.69806ms iteration = 21 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
6112025-01-10T05:15:52.779ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6122025-01-10T05:15:52.781ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
6132025-01-10T05:15:52.781ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
6142025-01-10T05:15:52.781ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
6152025-01-10T05:15:52.782ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
6162025-01-10T05:15:52.782ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
6172025-01-10T05:15:52.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
6182025-01-10T05:15:52.783ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
6192025-01-10T05:15:52.783ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
6202025-01-10T05:15:52.785ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 70f09b61-dde8-4da2-af2f-06138d5a975e dns_zone_name = control-plane.oxide.internal found_so_far = 51 version = 1
6212025-01-10T05:15:52.785ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
6222025-01-10T05:15:52.785ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-01-10 05:06:08.425967 UTC dns_group = internal generation = 1
6232025-01-10T05:15:52.785ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 128.431308ms iteration = 11 status = {"generation":1}
6242025-01-10T05:15:52.785ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
6252025-01-10T05:15:52.785ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
6262025-01-10T05:15:52.785ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
6272025-01-10T05:15:52.786ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
6282025-01-10T05:15:52.788ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
6292025-01-10T05:15:52.788ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
6302025-01-10T05:15:52.792ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6312025-01-10T05:15:52.792ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
6322025-01-10T05:15:52.793ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
6332025-01-10T05:15:52.793ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
6342025-01-10T05:15:52.794ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
6352025-01-10T05:15:52.794ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
6362025-01-10T05:15:52.794ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
6372025-01-10T05:15:52.796ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
6382025-01-10T05:15:52.796ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
6392025-01-10T05:15:52.797ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
6402025-01-10T05:15:52.797ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
6412025-01-10T05:15:52.799ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6422025-01-10T05:15:52.800ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
6432025-01-10T05:15:52.800ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
6442025-01-10T05:15:52.801ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6452025-01-10T05:15:52.805ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
6462025-01-10T05:15:52.805ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
6472025-01-10T05:15:52.805ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
6482025-01-10T05:15:52.807ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59, lookup_type: ById(a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59) } result = Ok(())
6492025-01-10T05:15:52.807ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
6502025-01-10T05:15:52.807ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
6512025-01-10T05:15:52.807ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 150.710351ms iteration = 21 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
6522025-01-10T05:15:52.808ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
6532025-01-10T05:15:52.808ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
6542025-01-10T05:15:52.808ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 152.167368ms iteration = 21 status = 0
6552025-01-10T05:15:52.810ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
6562025-01-10T05:15:52.810ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1111 zone_address = fd00:1122:3344:101::2
6572025-01-10T05:15:52.810ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
6582025-01-10T05:15:52.810ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
6592025-01-10T05:15:52.810ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1f291d0-cc7c-4d7b-945f-eed1683b37d2", "content-length": "26", "date": "Fri, 10 Jan 2025 05:15:52 GMT"} })
6602025-01-10T05:15:52.810ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1116 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
6612025-01-10T05:15:52.814ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6622025-01-10T05:15:52.815ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
6632025-01-10T05:15:52.815ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1148 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
6642025-01-10T05:15:52.818ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
6652025-01-10T05:15:52.818ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
6662025-01-10T05:15:52.818ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
6672025-01-10T05:15:52.823ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
6682025-01-10T05:15:52.824ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
6692025-01-10T05:15:52.826ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
6702025-01-10T05:15:52.826ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = bca3dac0-9831-415b-86b5-c6c6cc12ed56 dns_zone_name = oxide.test found_so_far = 0 version = 2
6712025-01-10T05:15:52.826ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
6722025-01-10T05:15:52.826ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
6732025-01-10T05:15:52.826ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
6742025-01-10T05:15:52.826ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
6752025-01-10T05:15:52.828ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6762025-01-10T05:15:52.828ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
6772025-01-10T05:15:52.828ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
6782025-01-10T05:15:52.831ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
6792025-01-10T05:15:52.831ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
6802025-01-10T05:15:52.831ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
6812025-01-10T05:15:52.831ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
6822025-01-10T05:15:52.831ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
6832025-01-10T05:15:52.833ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = bca3dac0-9831-415b-86b5-c6c6cc12ed56 dns_zone_name = oxide.test found_so_far = 1 version = 2
6842025-01-10T05:15:52.833ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
6852025-01-10T05:15:52.833ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-01-10 05:06:09.259013 UTC dns_group = external generation = 2
6862025-01-10T05:15:52.833ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 177.192571ms iteration = 11 status = {"generation":2}
6872025-01-10T05:15:52.834ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
6882025-01-10T05:15:52.835ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
6892025-01-10T05:15:52.848ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-01-10T05:05:45.360170Z, time_modified: 2025-01-10T05:05:45.360170Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 48a76ac8-2f94-4847-be0f-1cb4bfb661aa, db_silo: Silo { identity: SiloIdentity { id: 48a76ac8-2f94-4847-be0f-1cb4bfb661aa, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-01-10T05:06:09.192249Z, time_modified: 2025-01-10T05:06:09.192249Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "0de51367feb2d6f1959542dbb73fb728546b6d8dd71a0ec2fd5083120a87d20f" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 48a76ac8-2f94-4847-be0f-1cb4bfb661aa, db_silo: Silo { identity: SiloIdentity { id: 48a76ac8-2f94-4847-be0f-1cb4bfb661aa, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-01-10T05:06:09.192249Z, time_modified: 2025-01-10T05:06:09.192249Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "0de51367feb2d6f1959542dbb73fb728546b6d8dd71a0ec2fd5083120a87d20f" }] }) }
6902025-01-10T05:15:52.848ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 191.224336ms iteration = 11 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"48a76ac8-2f94-4847-be0f-1cb4bfb661aa","tls_certs":["0de51367feb2d6f1959542dbb73fb728546b6d8dd71a0ec2fd5083120a87d20f"]}},"default_endpoint":{"silo_id":"48a76ac8-2f94-4847-be0f-1cb4bfb661aa","tls_certs":["0de51367feb2d6f1959542dbb73fb728546b6d8dd71a0ec2fd5083120a87d20f"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
6912025-01-10T05:15:52.870ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 213.401065ms iteration = 22 status = {"physical_disks_added":0}
6922025-01-10T05:15:52.884ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
6932025-01-10T05:15:52.884ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 227.834114ms iteration = 11 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
6942025-01-10T05:15:52.896ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1111 zone_address = fd00:1122:3344:101::2
6952025-01-10T05:15:52.896ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
6962025-01-10T05:15:52.896ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae8eb355-4bf3-4105-9834-082b4d7c6ebe", "content-length": "26", "date": "Fri, 10 Jan 2025 05:15:52 GMT"} })
6972025-01-10T05:15:52.896ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1116 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
6982025-01-10T05:15:52.899ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1148 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
6992025-01-10T05:15:52.903ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
7002025-01-10T05:15:52.904ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
7012025-01-10T05:15:52.986ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1111 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 zone_address = fd00:1122:3344:101::2
7022025-01-10T05:15:52.986ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
7032025-01-10T05:15:52.989ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e7499bf-ec7a-4357-ae27-3b1c9b0651af", "content-length": "26", "date": "Fri, 10 Jan 2025 05:15:52 GMT"} })
7042025-01-10T05:15:52.989ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1116 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
7052025-01-10T05:15:52.992ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1148 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 79fde928-482f-41ab-aee3-3f2478145f08
7062025-01-10T05:15:52.998ZDEBGnexus (BackgroundTasks): begin collection background_task = inventory_collection
7072025-01-10T05:15:52.998ZDEBGnexus (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[fd00:1122:3344:101::2]:12225
7082025-01-10T05:15:52.998ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/ignition
7092025-01-10T05:15:52.998ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "b7fcae55-4567-4085-81d6-2439363627e6", "content-length": "476", "date": "Fri, 10 Jan 2025 05:15:52 GMT"} })
7102025-01-10T05:15:52.998ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0
7112025-01-10T05:15:52.998ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "943fd0a0-3fbc-4088-a194-b1633c5102c5", "content-length": "734", "date": "Fri, 10 Jan 2025 05:15:52 GMT"} })
7122025-01-10T05:15:52.999ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=0
7132025-01-10T05:15:52.999ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea4a1e85-3d45-4e5a-9d0e-833d67ec2e6d", "content-length": "111", "date": "Fri, 10 Jan 2025 05:15:52 GMT"} })
7142025-01-10T05:15:52.999ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=1
7152025-01-10T05:15:52.999ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "36561de1-c36d-4379-bf36-3b5dd7f0e090", "content-length": "111", "date": "Fri, 10 Jan 2025 05:15:52 GMT"} })
7162025-01-10T05:15:52.999ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=0
7172025-01-10T05:15:52.999ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5af346f-0cac-46f2-ae4c-944a5c2a429d", "content-length": "167", "date": "Fri, 10 Jan 2025 05:15:52 GMT"} })
7182025-01-10T05:15:52.999ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=1
7192025-01-10T05:15:53.000ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "99524baf-c43f-4425-aeb2-6b8483ed2369", "content-length": "167", "date": "Fri, 10 Jan 2025 05:15:52 GMT"} })
7202025-01-10T05:15:53.000ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=0
7212025-01-10T05:15:53.000ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dcb83fa6-2594-469d-ad17-c3720f54b729", "content-length": "175", "date": "Fri, 10 Jan 2025 05:15:52 GMT"} })
7222025-01-10T05:15:53.000ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=1
7232025-01-10T05:15:53.001ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "afcb4ec2-7af4-4171-b4f1-8db84f7390e7", "content-length": "175", "date": "Fri, 10 Jan 2025 05:15:52 GMT"} })
7242025-01-10T05:15:53.001ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cmpa
7252025-01-10T05:15:53.001ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "035a0762-f3db-4d86-a708-289dceb36bdd", "content-length": "702", "date": "Fri, 10 Jan 2025 05:15:52 GMT"} })
7262025-01-10T05:15:53.001ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa
7272025-01-10T05:15:53.001ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c43160cd-6519-46f5-a202-9b9f528b666e", "content-length": "727", "date": "Fri, 10 Jan 2025 05:15:52 GMT"} })
7282025-01-10T05:15:53.001ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa
7292025-01-10T05:15:53.001ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e102bcb-3c08-435f-b31d-df191af7ec9f", "content-length": "729", "date": "Fri, 10 Jan 2025 05:15:52 GMT"} })
7302025-01-10T05:15:53.001ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa
7312025-01-10T05:15:53.001ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "84807a39-ccf8-46e7-80f3-eca1537cf010", "content-length": "728", "date": "Fri, 10 Jan 2025 05:15:52 GMT"} })
7322025-01-10T05:15:53.001ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0
7332025-01-10T05:15:53.002ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4dc26136-343d-4e71-8e56-fa99ca974fd5", "content-length": "722", "date": "Fri, 10 Jan 2025 05:15:52 GMT"} })
7342025-01-10T05:15:53.002ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=0
7352025-01-10T05:15:53.002ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9f30367-c130-45a0-ad90-5865845ad997", "content-length": "109", "date": "Fri, 10 Jan 2025 05:15:52 GMT"} })
7362025-01-10T05:15:53.002ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=1
7372025-01-10T05:15:53.002ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9f1f129-e619-4384-98fa-106e36c89330", "content-length": "109", "date": "Fri, 10 Jan 2025 05:15:52 GMT"} })
7382025-01-10T05:15:53.002ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=0
7392025-01-10T05:15:53.003ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8bd31969-6ef6-48d9-a1ac-783b0d223de6", "content-length": "169", "date": "Fri, 10 Jan 2025 05:15:52 GMT"} })
7402025-01-10T05:15:53.003ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=1
7412025-01-10T05:15:53.003ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ee24da9-1a3a-419c-a363-63752cbf6dca", "content-length": "169", "date": "Fri, 10 Jan 2025 05:15:52 GMT"} })
7422025-01-10T05:15:53.003ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=0
7432025-01-10T05:15:53.003ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0358bee2-6412-46d5-b395-cbbab5739e16", "content-length": "178", "date": "Fri, 10 Jan 2025 05:15:52 GMT"} })
7442025-01-10T05:15:53.003ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=1
7452025-01-10T05:15:53.004ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a301276f-d7e4-4afa-b9ed-aafe1f8fabf3", "content-length": "178", "date": "Fri, 10 Jan 2025 05:15:52 GMT"} })
7462025-01-10T05:15:53.004ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cmpa
7472025-01-10T05:15:53.004ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0d0718a-4fc9-4b95-a766-0e08bdd14fad", "content-length": "702", "date": "Fri, 10 Jan 2025 05:15:52 GMT"} })
7482025-01-10T05:15:53.004ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa
7492025-01-10T05:15:53.004ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f017600-7768-4f79-b270-244eefc10f63", "content-length": "727", "date": "Fri, 10 Jan 2025 05:15:52 GMT"} })
7502025-01-10T05:15:53.004ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa
7512025-01-10T05:15:53.004ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "225871e0-29e7-40e5-80e8-9e553c3a1c6e", "content-length": "729", "date": "Fri, 10 Jan 2025 05:15:52 GMT"} })
7522025-01-10T05:15:53.004ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa
7532025-01-10T05:15:53.004ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d10b8263-24a3-4432-a15b-f10edcbcdd10", "content-length": "728", "date": "Fri, 10 Jan 2025 05:15:52 GMT"} })
7542025-01-10T05:15:53.004ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
7552025-01-10T05:15:53.005ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
7562025-01-10T05:15:53.005ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
7572025-01-10T05:15:53.006ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e8a3347-f5bd-46f9-a460-cb85bb0c1914", "content-length": "1", "date": "Fri, 10 Jan 2025 05:15:53 GMT"} })
7582025-01-10T05:15:53.008ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
7592025-01-10T05:15:53.009ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
7602025-01-10T05:15:53.010ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
7612025-01-10T05:15:53.010ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "bd4f72d0-9896-4529-912e-7f4c2426eeb8", "date": "Fri, 10 Jan 2025 05:15:52 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
7622025-01-10T05:15:53.010ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
7632025-01-10T05:15:53.012ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "8fffacb6-ee27-4aba-a64d-0e560965e645", "date": "Fri, 10 Jan 2025 05:15:52 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
7642025-01-10T05:15:53.012ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
7652025-01-10T05:15:53.018ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
7662025-01-10T05:15:53.018ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 361.663437ms iteration = 13 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
7672025-01-10T05:15:53.033ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
7682025-01-10T05:15:53.054ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 397.915373ms iteration = 21 status = {}
7692025-01-10T05:15:53.056ZINFOnexus (BackgroundTasks): VPC route manager sled a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
7702025-01-10T05:15:53.056ZDEBGnexus (BackgroundTasks): client request SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
7712025-01-10T05:15:53.059ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7722025-01-10T05:15:53.059ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7732025-01-10T05:15:53.059ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7742025-01-10T05:15:53.060ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7752025-01-10T05:15:53.060ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 403.935299ms iteration = 21 status = null
7762025-01-10T05:15:53.061ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7772025-01-10T05:15:53.061ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7782025-01-10T05:15:53.062ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
7792025-01-10T05:15:53.062ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
7802025-01-10T05:15:53.062ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
7812025-01-10T05:15:53.063ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7822025-01-10T05:15:53.063ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
7832025-01-10T05:15:53.063ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
7842025-01-10T05:15:53.063ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
7852025-01-10T05:15:53.064ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7862025-01-10T05:15:53.065ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7872025-01-10T05:15:53.066ZDEBGnexus (BackgroundTasks): client response SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d1c3d6a-3901-46ce-bddd-19b0d2cb9ce4", "content-length": "372", "date": "Fri, 10 Jan 2025 05:15:52 GMT"} })
7882025-01-10T05:15:53.066ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
7892025-01-10T05:15:53.066ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
7902025-01-10T05:15:53.066ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
7912025-01-10T05:15:53.066ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
7922025-01-10T05:15:53.067ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
7932025-01-10T05:15:53.067ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
7942025-01-10T05:15:53.067ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7952025-01-10T05:15:53.067ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7962025-01-10T05:15:53.069ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
7972025-01-10T05:15:53.069ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
7982025-01-10T05:15:53.069ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
7992025-01-10T05:15:53.069ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8002025-01-10T05:15:53.070ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8012025-01-10T05:15:53.071ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8022025-01-10T05:15:53.071ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8032025-01-10T05:15:53.073ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8042025-01-10T05:15:53.076ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
8052025-01-10T05:15:53.076ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8062025-01-10T05:15:53.076ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8072025-01-10T05:15:53.083ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
8082025-01-10T05:15:53.083ZDEBGnexus (BackgroundTasks): client request SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
8092025-01-10T05:15:53.085ZDEBGnexus (BackgroundTasks): client response SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "dd9d2fd0-4e7f-415d-9300-ad50e1158db8", "date": "Fri, 10 Jan 2025 05:15:53 GMT"} })
8102025-01-10T05:15:53.085ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
8112025-01-10T05:15:53.085ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
8122025-01-10T05:15:53.129ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
8132025-01-10T05:15:53.129ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "24312635-8f09-4a05-b5dc-42d80fcc8420", "date": "Fri, 10 Jan 2025 05:15:52 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
8142025-01-10T05:15:53.129ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
8152025-01-10T05:15:53.131ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "c13e5c22-e0bd-4e15-acc5-0ff6b316b438", "date": "Fri, 10 Jan 2025 05:15:52 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
8162025-01-10T05:15:53.131ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
8172025-01-10T05:15:53.133ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "f1038560-a825-47e2-b9a8-f23d5ff34e44", "date": "Fri, 10 Jan 2025 05:15:52 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
8182025-01-10T05:15:53.133ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
8192025-01-10T05:15:53.135ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
8202025-01-10T05:15:53.135ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 478.602772ms iteration = 13 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
8212025-01-10T05:15:53.173ZDEBGnexus (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[fd00:1122:3344:101::1]:12345
8222025-01-10T05:15:53.173ZDEBGnexus (BackgroundTasks): client request SledAgent = http://[fd00:1122:3344:101::1]:12345 background_task = inventory_collection body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/inventory
8232025-01-10T05:15:53.174ZDEBGnexus (BackgroundTasks): client request SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
8242025-01-10T05:15:53.174ZDEBGnexus (BackgroundTasks): client response SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "62e65c51-645b-4e83-adb8-070502bb7acc", "content-length": "691", "date": "Fri, 10 Jan 2025 05:15:53 GMT"} })
8252025-01-10T05:15:53.174ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
8262025-01-10T05:15:53.174ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
8272025-01-10T05:15:53.174ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
8282025-01-10T05:15:53.176ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 519.682135ms iteration = 21 status = {}
8292025-01-10T05:15:53.177ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
8302025-01-10T05:15:53.177ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
8312025-01-10T05:15:53.180ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
8322025-01-10T05:15:53.180ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
8332025-01-10T05:15:53.181ZDEBGnexus (service-firewall-rule-progator): resolved names for firewall rules background_task = service_firewall_rule_propagation instance_interfaces = {} subnet_interfaces = {Name("external-dns"): [NetworkInterface { id: 677a3534-34ba-4230-bb45-84a07cadda91, kind: Service { id: 95b462f2-3c97-4463-9c08-5404518f33d2 }, name: Name("external-dns-95b462f2-3c97-4463-9c08-5404518f33d2"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 167, 9])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: a7579baa-4dc6-41c4-aa40-d1574c336fc8, kind: Service { id: 30a043aa-79b8-40f0-889b-b1aa4686508a }, name: Name("external-dns-30a043aa-79b8-40f0-889b-b1aa4686508a"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 227, 129])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], Name("nexus"): [NetworkInterface { id: 6b77252c-3753-4653-a12d-8d9cf4b8b034, kind: Service { id: 3ca89dd1-7ada-4fe6-8600-a84d41dfa9d8 }, name: Name("nexus-3ca89dd1-7ada-4fe6-8600-a84d41dfa9d8"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 239, 193])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 88ecb166-c6e5-4bcc-886a-a1c3a146d28d, kind: Service { id: a5a77b64-bc6f-4038-bc69-a5f388ab2baa }, name: Name("nexus-a5a77b64-bc6f-4038-bc69-a5f388ab2baa"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 240, 231])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: bf07848a-01b0-4d7b-822b-aef5c0a3de3f, kind: Service { id: 36b3525a-4d18-4be7-8420-a6b90a955a44 }, name: Name("nexus-36b3525a-4d18-4be7-8420-a6b90a955a44"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 194, 198])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }]} subnet_networks = {Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })], Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })]} vpc_interfaces = {}
8342025-01-10T05:15:53.181ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8352025-01-10T05:15:53.181ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8362025-01-10T05:15:53.183ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8372025-01-10T05:15:53.183ZINFOnexus (BackgroundTasks): VPC route manager sled a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
8382025-01-10T05:15:53.183ZDEBGnexus (BackgroundTasks): client request SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
8392025-01-10T05:15:53.185ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Fleet result = Ok(())
8402025-01-10T05:15:53.185ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8412025-01-10T05:15:53.185ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8422025-01-10T05:15:53.187ZDEBGnexus (service-firewall-rule-progator): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified. background_task = service_firewall_rule_propagation
8432025-01-10T05:15:53.187ZDEBGnexus (service-firewall-rule-progator): resolved firewall rules for sled agents background_task = service_firewall_rule_propagation sled_agent_rules = [ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("53")]), filter_protocols: Some([Udp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 677a3534-34ba-4230-bb45-84a07cadda91, kind: Service { id: 95b462f2-3c97-4463-9c08-5404518f33d2 }, name: Name("external-dns-95b462f2-3c97-4463-9c08-5404518f33d2"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 167, 9])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: a7579baa-4dc6-41c4-aa40-d1574c336fc8, kind: Service { id: 30a043aa-79b8-40f0-889b-b1aa4686508a }, name: Name("external-dns-30a043aa-79b8-40f0-889b-b1aa4686508a"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 227, 129])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }, ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("80"), L4PortRange("443")]), filter_protocols: Some([Tcp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 6b77252c-3753-4653-a12d-8d9cf4b8b034, kind: Service { id: 3ca89dd1-7ada-4fe6-8600-a84d41dfa9d8 }, name: Name("nexus-3ca89dd1-7ada-4fe6-8600-a84d41dfa9d8"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 239, 193])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 88ecb166-c6e5-4bcc-886a-a1c3a146d28d, kind: Service { id: a5a77b64-bc6f-4038-bc69-a5f388ab2baa }, name: Name("nexus-a5a77b64-bc6f-4038-bc69-a5f388ab2baa"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 240, 231])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: bf07848a-01b0-4d7b-822b-aef5c0a3de3f, kind: Service { id: 36b3525a-4d18-4be7-8420-a6b90a955a44 }, name: Name("nexus-36b3525a-4d18-4be7-8420-a6b90a955a44"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 194, 198])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }]
8442025-01-10T05:15:53.187ZDEBGnexus (service-firewall-rule-progator): resolved 2 rules for sleds background_task = service_firewall_rule_propagation
8452025-01-10T05:15:53.187ZDEBGnexus (BackgroundTasks): client response SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "a158bcb0-2d7c-429b-ba70-af386ad5166d", "date": "Fri, 10 Jan 2025 05:15:52 GMT"} })
8462025-01-10T05:15:53.189ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 532.325096ms iteration = 21 status = {}
8472025-01-10T05:15:53.213ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
8482025-01-10T05:15:53.213ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
8492025-01-10T05:15:53.215ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 5263359f-a80d-45e0-9595-0b15b904e78f, rack_id: 79fde928-482f-41ab-aee3-3f2478145f08, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(77858eb2-1069-4750-be5e-f3dafe20063a) } rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 switch_location = Switch0
8502025-01-10T05:15:53.215ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
8512025-01-10T05:15:53.215ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
8522025-01-10T05:15:53.224ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
8532025-01-10T05:15:53.224ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "b671242f-fbc1-4e4f-a307-57f7ef5ae105", "content-length": "134", "date": "Fri, 10 Jan 2025 05:15:53 GMT"} })
8542025-01-10T05:15:53.224ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
8552025-01-10T05:15:53.224ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 77858eb2-1069-4750-be5e-f3dafe20063a
8562025-01-10T05:15:53.225ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
8572025-01-10T05:15:53.225ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
8582025-01-10T05:15:53.225ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
8592025-01-10T05:15:53.225ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
8602025-01-10T05:15:53.225ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
8612025-01-10T05:15:53.225ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
8622025-01-10T05:15:53.225ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
8632025-01-10T05:15:53.225ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
8642025-01-10T05:15:53.225ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08
8652025-01-10T05:15:53.225ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
8662025-01-10T05:15:53.225ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
8672025-01-10T05:15:53.228ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
8682025-01-10T05:15:53.228ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "257912ec-09e6-45c8-9ef0-8cca54b19861", "content-length": "2", "date": "Fri, 10 Jan 2025 05:15:53 GMT"} })
8692025-01-10T05:15:53.228ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
8702025-01-10T05:15:53.228ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e1e44a9-cb8f-482a-9ff2-4af488bb1517", "content-length": "52", "date": "Fri, 10 Jan 2025 05:15:53 GMT"} })
8712025-01-10T05:15:53.228ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 switch_location = Switch0
8722025-01-10T05:15:53.228ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
8732025-01-10T05:15:53.245ZDEBGnexus (service-firewall-rule-progator): resolved sleds for vpc oxide-services background_task = service_firewall_rule_propagation vpc_to_sled = [Sled { identity: SledIdentity { id: a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59, time_created: 2025-01-10T05:06:06.242215Z, time_modified: 2025-01-10T05:06:06.242215Z }, time_deleted: None, rcgen: Generation(Generation(19)), rack_id: 79fde928-482f-41ab-aee3-3f2478145f08, is_scrimlet: true, serial_number: "sock", part_number: "i86pc", revision: SqlU32(0), usable_hardware_threads: SqlU32(48), usable_physical_ram: ByteCount(ByteCount(102952026112)), reservoir_size: ByteCount(ByteCount(51474595840)), ip: fd00:1122:3344:101::1, port: SqlU16(12345), last_used_address: fd00:1122:3344:101::ffff, policy: InService, state: Active, sled_agent_gen: Generation(Generation(1)) }]
8742025-01-10T05:15:53.245ZDEBGnexus (service-firewall-rule-progator): sending firewall rules to sled agents background_task = service_firewall_rule_propagation
8752025-01-10T05:15:53.245ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8762025-01-10T05:15:53.245ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8772025-01-10T05:15:53.247ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8782025-01-10T05:15:53.248ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8792025-01-10T05:15:53.249ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8802025-01-10T05:15:53.252ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Sled { parent: Fleet, key: a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59, lookup_type: ById(a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59) } result = Ok(())
8812025-01-10T05:15:53.287ZDEBGnexus (service-firewall-rule-progator): client request SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = service_firewall_rule_propagation body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules
8822025-01-10T05:15:53.294ZDEBGnexus (service-firewall-rule-progator): client response SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = service_firewall_rule_propagation result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules", status: 204, headers: {"x-request-id": "4897b593-7a88-41c7-8e2a-f4172409c6ab", "date": "Fri, 10 Jan 2025 05:15:53 GMT"} })
8832025-01-10T05:15:53.295ZINFOnexus (service-firewall-rule-progator): updated firewall rules on 1 sleds background_task = service_firewall_rule_propagation file = nexus/networking/src/firewall_rules.rs:496
8842025-01-10T05:15:53.295ZDEBGnexus (service-firewall-rule-progator): successfully propagated service firewall rules background_task = service_firewall_rule_propagation elapsed = 638.286644ms
8852025-01-10T05:15:53.295ZDEBGnexus (BackgroundTasks): activation complete background_task = service_firewall_rule_propagation elapsed = 638.295474ms iteration = 3 status = {}
8862025-01-10T05:15:53.351ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "1f43068e-c292-4e7a-b5cb-2f626b3decac", "date": "Fri, 10 Jan 2025 05:15:52 GMT"} })
8872025-01-10T05:15:53.351ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 switch = Switch0
8882025-01-10T05:15:53.351ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
8892025-01-10T05:15:53.351ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "6629c1b5-1bec-4da5-8be8-1bc1e076b0ca", "content-length": "682", "date": "Fri, 10 Jan 2025 05:15:53 GMT"} })
8902025-01-10T05:15:53.352ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
8912025-01-10T05:15:53.352ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
8922025-01-10T05:15:53.360ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
8932025-01-10T05:15:53.360ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
8942025-01-10T05:15:53.363ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
8952025-01-10T05:15:53.364ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
8962025-01-10T05:15:53.366ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
8972025-01-10T05:15:53.366ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
8982025-01-10T05:15:53.373ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 716.322127ms iteration = 21 status = {}
8992025-01-10T05:15:53.538ZDEBGnexus (BackgroundTasks): client response SledAgent = http://[fd00:1122:3344:101::1]:12345 background_task = inventory_collection result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e6dd43f-d147-4552-b781-793a4da1bd1f", "content-length": "30795", "date": "Fri, 10 Jan 2025 05:15:53 GMT"} })
9002025-01-10T05:15:53.553ZDEBGnexus (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
9012025-01-10T05:15:53.553ZDEBGnexus (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
9022025-01-10T05:15:53.553ZDEBGnexus (BackgroundTasks): finished collection background_task = inventory_collection
9032025-01-10T05:15:53.557ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
9042025-01-10T05:15:53.558ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
9052025-01-10T05:15:53.559ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9062025-01-10T05:15:53.561ZDEBGnexus (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
9072025-01-10T05:15:53.561ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
9082025-01-10T05:15:53.562ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
9092025-01-10T05:15:53.738ZINFOnexus (BackgroundTasks): inserted inventory collection background_task = inventory_collection collection_id = e0105262-2b7c-4092-b7e4-2d0fbfa6c40b file = nexus/db-queries/src/db/datastore/inventory.rs:1010
9102025-01-10T05:15:53.738ZDEBGnexus (BackgroundTasks): inventory collection complete background_task = inventory_collection collection_id = e0105262-2b7c-4092-b7e4-2d0fbfa6c40b time_started = 2025-01-10 05:15:52.998246 UTC
9112025-01-10T05:15:53.739ZDEBGnexus (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.082077107s iteration = 2 status = {"collection_id":"e0105262-2b7c-4092-b7e4-2d0fbfa6c40b","time_done":"2025-01-10 05:15:53.555863 UTC","time_started":"2025-01-10 05:15:52.998246 UTC"}
9122025-01-10T05:15:53.739ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 23 reason = Dependency
9132025-01-10T05:15:53.739ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
9142025-01-10T05:15:53.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
9152025-01-10T05:15:53.740ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9162025-01-10T05:15:53.742ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
9172025-01-10T05:15:53.742ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
9182025-01-10T05:15:53.742ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
9192025-01-10T05:15:53.746ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
9202025-01-10T05:15:53.746ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
9212025-01-10T05:15:53.748ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9222025-01-10T05:15:53.750ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
9232025-01-10T05:15:53.750ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
9242025-01-10T05:15:53.751ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
9252025-01-10T05:15:53.775ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 36.458134ms iteration = 23 status = {"physical_disks_added":0}
9262025-01-10T05:15:57.549ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 265 local_addr = [fd00:1122:3344:101::c]:57161 method = GET remote_addr = [fd00:1122:3344:101::d]:46894 req_id = 425c7d30-0e6e-4f24-8277-1339106ac09a response_code = 200 uri = /3ca89dd1-7ada-4fe6-8600-a84d41dfa9d8
9272025-01-10T05:16:02.657ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 62 reason = Timeout
9282025-01-10T05:16:02.657ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 62 reason = Timeout
9292025-01-10T05:16:02.657ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9302025-01-10T05:16:02.657ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
9312025-01-10T05:16:02.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9322025-01-10T05:16:02.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
9332025-01-10T05:16:02.659ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9342025-01-10T05:16:02.660ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
9352025-01-10T05:16:02.660ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
9362025-01-10T05:16:02.662ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.653891ms iteration = 62 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
9372025-01-10T05:16:02.662ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
9382025-01-10T05:16:02.662ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9392025-01-10T05:16:02.662ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9402025-01-10T05:16:02.664ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9412025-01-10T05:16:02.664ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9422025-01-10T05:16:02.666ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9432025-01-10T05:16:02.669ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 43ea24da-5730-4a85-9a8e-76459f9b69b1, lookup_type: ById(43ea24da-5730-4a85-9a8e-76459f9b69b1) } result = Ok(())
9442025-01-10T05:16:02.669ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9452025-01-10T05:16:02.669ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9462025-01-10T05:16:02.677ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 original_time_created = 2025-01-10 05:05:43.319091 UTC target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 time_created = 2025-01-10 05:05:43.319091 UTC
9472025-01-10T05:16:02.677ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.397152ms iteration = 62 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"43ea24da-5730-4a85-9a8e-76459f9b69b1","time_created":"2025-01-10T05:05:43.319091Z"}
9482025-01-10T05:16:07.549ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 295 local_addr = [fd00:1122:3344:101::c]:57161 method = GET remote_addr = [fd00:1122:3344:101::d]:46894 req_id = 4addb66b-8eb3-44c1-9244-c08d447a0ea8 response_code = 200 uri = /3ca89dd1-7ada-4fe6-8600-a84d41dfa9d8
9492025-01-10T05:16:12.656ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 63 reason = Timeout
9502025-01-10T05:16:12.656ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 63 reason = Timeout
9512025-01-10T05:16:12.656ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9522025-01-10T05:16:12.656ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
9532025-01-10T05:16:12.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9542025-01-10T05:16:12.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
9552025-01-10T05:16:12.658ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9562025-01-10T05:16:12.659ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
9572025-01-10T05:16:12.659ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
9582025-01-10T05:16:12.661ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.857968ms iteration = 63 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
9592025-01-10T05:16:12.661ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
9602025-01-10T05:16:12.661ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9612025-01-10T05:16:12.661ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9622025-01-10T05:16:12.663ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9632025-01-10T05:16:12.663ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9642025-01-10T05:16:12.664ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9652025-01-10T05:16:12.667ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 43ea24da-5730-4a85-9a8e-76459f9b69b1, lookup_type: ById(43ea24da-5730-4a85-9a8e-76459f9b69b1) } result = Ok(())
9662025-01-10T05:16:12.667ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9672025-01-10T05:16:12.667ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9682025-01-10T05:16:12.676ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 original_time_created = 2025-01-10 05:05:43.319091 UTC target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 time_created = 2025-01-10 05:05:43.319091 UTC
9692025-01-10T05:16:12.676ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.372011ms iteration = 63 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"43ea24da-5730-4a85-9a8e-76459f9b69b1","time_created":"2025-01-10T05:05:43.319091Z"}
9702025-01-10T05:16:17.549ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 276 local_addr = [fd00:1122:3344:101::c]:57161 method = GET remote_addr = [fd00:1122:3344:101::d]:46894 req_id = 175af93b-88b5-4aea-aa5a-bd1ef64e8e23 response_code = 200 uri = /3ca89dd1-7ada-4fe6-8600-a84d41dfa9d8
9712025-01-10T05:16:22.656ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 64 reason = Timeout
9722025-01-10T05:16:22.656ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 64 reason = Timeout
9732025-01-10T05:16:22.656ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9742025-01-10T05:16:22.656ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
9752025-01-10T05:16:22.656ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 22 reason = Timeout
9762025-01-10T05:16:22.656ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
9772025-01-10T05:16:22.656ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 24 reason = Timeout
9782025-01-10T05:16:22.656ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 22 reason = Timeout
9792025-01-10T05:16:22.656ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
9802025-01-10T05:16:22.656ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9812025-01-10T05:16:22.656ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 22 reason = Timeout
9822025-01-10T05:16:22.656ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 22 reason = Timeout
9832025-01-10T05:16:22.656ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
9842025-01-10T05:16:22.656ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
9852025-01-10T05:16:22.656ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 22 reason = Timeout
9862025-01-10T05:16:22.656ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
9872025-01-10T05:16:22.656ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 22 reason = Timeout
9882025-01-10T05:16:22.656ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 22 reason = Timeout
9892025-01-10T05:16:22.656ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
9902025-01-10T05:16:22.656ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 22 reason = Timeout
9912025-01-10T05:16:22.656ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
9922025-01-10T05:16:22.656ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
9932025-01-10T05:16:22.656ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 22 reason = Timeout
9942025-01-10T05:16:22.656ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 22 reason = Timeout
9952025-01-10T05:16:22.656ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
9962025-01-10T05:16:22.656ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
9972025-01-10T05:16:22.656ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 22 reason = Timeout
9982025-01-10T05:16:22.656ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
9992025-01-10T05:16:22.656ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 22 reason = Timeout
10002025-01-10T05:16:22.656ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
10012025-01-10T05:16:22.656ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
10022025-01-10T05:16:22.656ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 22 reason = Timeout
10032025-01-10T05:16:22.656ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
10042025-01-10T05:16:22.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10052025-01-10T05:16:22.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
10062025-01-10T05:16:22.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
10072025-01-10T05:16:22.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
10082025-01-10T05:16:22.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
10092025-01-10T05:16:22.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
10102025-01-10T05:16:22.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
10112025-01-10T05:16:22.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
10122025-01-10T05:16:22.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
10132025-01-10T05:16:22.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
10142025-01-10T05:16:22.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
10152025-01-10T05:16:22.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
10162025-01-10T05:16:22.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
10172025-01-10T05:16:22.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
10182025-01-10T05:16:22.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
10192025-01-10T05:16:22.658ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 22 reason = Timeout
10202025-01-10T05:16:22.658ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
10212025-01-10T05:16:22.658ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
10222025-01-10T05:16:22.659ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10232025-01-10T05:16:22.660ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.492503ms iteration = 22 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
10242025-01-10T05:16:22.660ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10252025-01-10T05:16:22.660ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
10262025-01-10T05:16:22.660ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
10272025-01-10T05:16:22.661ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 26993, tv_nsec: 168526293 } })
10282025-01-10T05:16:22.662ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1103
10292025-01-10T05:16:22.662ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
10302025-01-10T05:16:22.662ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10312025-01-10T05:16:22.662ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10322025-01-10T05:16:22.663ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
10332025-01-10T05:16:22.663ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
10342025-01-10T05:16:22.663ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
10352025-01-10T05:16:22.664ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10362025-01-10T05:16:22.665ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10372025-01-10T05:16:22.666ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 26993, tv_nsec: 172681118 } })
10382025-01-10T05:16:22.666ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1103
10392025-01-10T05:16:22.667ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
10402025-01-10T05:16:22.667ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
10412025-01-10T05:16:22.667ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
10422025-01-10T05:16:22.667ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
10432025-01-10T05:16:22.667ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
10442025-01-10T05:16:22.668ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
10452025-01-10T05:16:22.671ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10462025-01-10T05:16:22.671ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10472025-01-10T05:16:22.672ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10482025-01-10T05:16:22.672ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10492025-01-10T05:16:22.672ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
10502025-01-10T05:16:22.672ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
10512025-01-10T05:16:22.673ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
10522025-01-10T05:16:22.673ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
10532025-01-10T05:16:22.673ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
10542025-01-10T05:16:22.675ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
10552025-01-10T05:16:22.675ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
10562025-01-10T05:16:22.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
10572025-01-10T05:16:22.676ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
10582025-01-10T05:16:22.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
10592025-01-10T05:16:22.680ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10602025-01-10T05:16:22.680ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
10612025-01-10T05:16:22.680ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
10622025-01-10T05:16:22.681ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 26993, tv_nsec: 187880611 } })
10632025-01-10T05:16:22.681ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1103 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08
10642025-01-10T05:16:22.681ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 25.292951ms iteration = 22 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
10652025-01-10T05:16:22.682ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10662025-01-10T05:16:22.685ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 43ea24da-5730-4a85-9a8e-76459f9b69b1, lookup_type: ById(43ea24da-5730-4a85-9a8e-76459f9b69b1) } result = Ok(())
10672025-01-10T05:16:22.685ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10682025-01-10T05:16:22.685ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10692025-01-10T05:16:22.685ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
10702025-01-10T05:16:22.685ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
10712025-01-10T05:16:22.686ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
10722025-01-10T05:16:22.689ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10732025-01-10T05:16:22.691ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
10742025-01-10T05:16:22.691ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
10752025-01-10T05:16:22.691ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
10762025-01-10T05:16:22.693ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
10772025-01-10T05:16:22.693ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
10782025-01-10T05:16:22.701ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 original_time_created = 2025-01-10 05:05:43.319091 UTC target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 time_created = 2025-01-10 05:05:43.319091 UTC
10792025-01-10T05:16:22.701ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 44.807414ms iteration = 64 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"43ea24da-5730-4a85-9a8e-76459f9b69b1","time_created":"2025-01-10T05:05:43.319091Z"}
10802025-01-10T05:16:22.703ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
10812025-01-10T05:16:22.703ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
10822025-01-10T05:16:22.703ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
10832025-01-10T05:16:22.703ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
10842025-01-10T05:16:22.704ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 48.279696ms iteration = 22 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
10852025-01-10T05:16:22.713ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
10862025-01-10T05:16:22.713ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
10872025-01-10T05:16:22.714ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
10882025-01-10T05:16:22.714ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
10892025-01-10T05:16:22.714ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 57.936831ms iteration = 22 status = {"errors":[],"garbage_collect_requested":[]}
10902025-01-10T05:16:22.716ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
10912025-01-10T05:16:22.716ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
10922025-01-10T05:16:22.716ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 60.312797ms iteration = 22 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
10932025-01-10T05:16:22.717ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10942025-01-10T05:16:22.717ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 59.196092ms iteration = 22 status = {"errors":[],"finish_invoked_ok":[]}
10952025-01-10T05:16:22.718ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
10962025-01-10T05:16:22.718ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
10972025-01-10T05:16:22.718ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
10982025-01-10T05:16:22.718ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
10992025-01-10T05:16:22.720ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59, lookup_type: ById(a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59) } result = Ok(())
11002025-01-10T05:16:22.720ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
11012025-01-10T05:16:22.721ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
11022025-01-10T05:16:22.723ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 66.756006ms iteration = 22 status = 0
11032025-01-10T05:16:22.723ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
11042025-01-10T05:16:22.723ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
11052025-01-10T05:16:22.723ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
11062025-01-10T05:16:22.723ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
11072025-01-10T05:16:22.725ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
11082025-01-10T05:16:22.725ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
11092025-01-10T05:16:22.725ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 69.03627ms iteration = 22 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
11102025-01-10T05:16:22.729ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
11112025-01-10T05:16:22.729ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 72.664508ms iteration = 22 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
11122025-01-10T05:16:22.735ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 79.370516ms iteration = 64 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
11132025-01-10T05:16:22.743ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1111 zone_address = fd00:1122:3344:101::2
11142025-01-10T05:16:22.743ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
11152025-01-10T05:16:22.743ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "1999dc1f-3acf-4848-8e4e-1be3133729b3", "content-length": "26", "date": "Fri, 10 Jan 2025 05:16:22 GMT"} })
11162025-01-10T05:16:22.743ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1116 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
11172025-01-10T05:16:22.745ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1148 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
11182025-01-10T05:16:22.746ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1111 zone_address = fd00:1122:3344:101::2
11192025-01-10T05:16:22.746ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
11202025-01-10T05:16:22.747ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ac511e3-bcba-410d-ac54-19c76e59cf94", "content-length": "26", "date": "Fri, 10 Jan 2025 05:16:22 GMT"} })
11212025-01-10T05:16:22.747ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1116 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
11222025-01-10T05:16:22.748ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1148 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
11232025-01-10T05:16:22.765ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1111 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 zone_address = fd00:1122:3344:101::2
11242025-01-10T05:16:22.765ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
11252025-01-10T05:16:22.765ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "6714dc28-981b-46ae-a3f0-eff7a1316ee9", "content-length": "26", "date": "Fri, 10 Jan 2025 05:16:22 GMT"} })
11262025-01-10T05:16:22.765ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1116 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
11272025-01-10T05:16:22.767ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1148 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 79fde928-482f-41ab-aee3-3f2478145f08
11282025-01-10T05:16:22.797ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11292025-01-10T05:16:22.797ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11302025-01-10T05:16:22.799ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11312025-01-10T05:16:22.801ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
11322025-01-10T05:16:22.801ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11332025-01-10T05:16:22.801ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11342025-01-10T05:16:22.801ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 145.334247ms iteration = 24 status = {"physical_disks_added":0}
11352025-01-10T05:16:22.848ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
11362025-01-10T05:16:22.848ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f223b09-07a6-4133-a27c-bd9d17278ad0", "content-length": "1", "date": "Fri, 10 Jan 2025 05:16:22 GMT"} })
11372025-01-10T05:16:22.850ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
11382025-01-10T05:16:22.850ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
11392025-01-10T05:16:22.860ZINFOnexus (BackgroundTasks): VPC route manager sled a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
11402025-01-10T05:16:22.860ZDEBGnexus (BackgroundTasks): client request SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
11412025-01-10T05:16:22.861ZDEBGnexus (BackgroundTasks): client response SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f6ad001-2ee8-4a39-800c-b4bedd9de811", "content-length": "372", "date": "Fri, 10 Jan 2025 05:16:22 GMT"} })
11422025-01-10T05:16:22.861ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
11432025-01-10T05:16:22.861ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
11442025-01-10T05:16:22.861ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 205.108016ms iteration = 22 status = {}
11452025-01-10T05:16:22.863ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 207.036399ms iteration = 22 status = null
11462025-01-10T05:16:22.892ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11472025-01-10T05:16:22.894ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
11482025-01-10T05:16:22.894ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11492025-01-10T05:16:22.894ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11502025-01-10T05:16:22.906ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
11512025-01-10T05:16:22.906ZDEBGnexus (BackgroundTasks): client request SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
11522025-01-10T05:16:22.906ZDEBGnexus (BackgroundTasks): client response SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "813702cf-1969-4499-b157-5f9d24839cbc", "date": "Fri, 10 Jan 2025 05:16:22 GMT"} })
11532025-01-10T05:16:22.906ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
11542025-01-10T05:16:22.907ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
11552025-01-10T05:16:22.909ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
11562025-01-10T05:16:22.909ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
11572025-01-10T05:16:22.912ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
11582025-01-10T05:16:22.912ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
11592025-01-10T05:16:22.950ZDEBGnexus (BackgroundTasks): client request SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
11602025-01-10T05:16:22.951ZDEBGnexus (BackgroundTasks): client response SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f335488-87b1-4d3d-81a2-a9f148df9ab4", "content-length": "691", "date": "Fri, 10 Jan 2025 05:16:22 GMT"} })
11612025-01-10T05:16:22.951ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
11622025-01-10T05:16:22.951ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
11632025-01-10T05:16:22.951ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
11642025-01-10T05:16:22.952ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 295.685056ms iteration = 22 status = {}
11652025-01-10T05:16:22.954ZINFOnexus (BackgroundTasks): VPC route manager sled a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
11662025-01-10T05:16:22.954ZDEBGnexus (BackgroundTasks): client request SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
11672025-01-10T05:16:22.954ZDEBGnexus (BackgroundTasks): client response SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "7a7a9c6b-029d-4e2e-ac33-4bf3472f9816", "date": "Fri, 10 Jan 2025 05:16:22 GMT"} })
11682025-01-10T05:16:22.956ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 299.660836ms iteration = 22 status = {}
11692025-01-10T05:16:22.970ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
11702025-01-10T05:16:22.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
11712025-01-10T05:16:22.972ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 5263359f-a80d-45e0-9595-0b15b904e78f, rack_id: 79fde928-482f-41ab-aee3-3f2478145f08, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(77858eb2-1069-4750-be5e-f3dafe20063a) } rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 switch_location = Switch0
11722025-01-10T05:16:22.972ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
11732025-01-10T05:16:22.972ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
11742025-01-10T05:16:22.982ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
11752025-01-10T05:16:22.982ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8e75d84-0659-406d-960f-48ce674d1c23", "content-length": "134", "date": "Fri, 10 Jan 2025 05:16:22 GMT"} })
11762025-01-10T05:16:22.982ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
11772025-01-10T05:16:22.982ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 77858eb2-1069-4750-be5e-f3dafe20063a
11782025-01-10T05:16:22.983ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
11792025-01-10T05:16:22.983ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
11802025-01-10T05:16:22.983ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
11812025-01-10T05:16:22.983ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
11822025-01-10T05:16:22.983ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
11832025-01-10T05:16:22.983ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
11842025-01-10T05:16:22.983ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
11852025-01-10T05:16:22.983ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
11862025-01-10T05:16:22.983ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08
11872025-01-10T05:16:22.983ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
11882025-01-10T05:16:22.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
11892025-01-10T05:16:22.984ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
11902025-01-10T05:16:22.985ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5da9f7f-0391-4c94-8ced-a36642b048c8", "content-length": "2", "date": "Fri, 10 Jan 2025 05:16:22 GMT"} })
11912025-01-10T05:16:22.985ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
11922025-01-10T05:16:22.985ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "fba8a932-177b-4c46-9b5c-95e0a8af3471", "content-length": "52", "date": "Fri, 10 Jan 2025 05:16:22 GMT"} })
11932025-01-10T05:16:22.985ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 switch_location = Switch0
11942025-01-10T05:16:22.985ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
11952025-01-10T05:16:23.074ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "fcb3063e-07a7-4759-80a1-916824e8854a", "date": "Fri, 10 Jan 2025 05:16:22 GMT"} })
11962025-01-10T05:16:23.074ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 switch = Switch0
11972025-01-10T05:16:23.074ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
11982025-01-10T05:16:23.074ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9e248a3-5f54-469c-bbda-c55aa4f5a0bf", "content-length": "682", "date": "Fri, 10 Jan 2025 05:16:22 GMT"} })
11992025-01-10T05:16:23.074ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
12002025-01-10T05:16:23.075ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
12012025-01-10T05:16:23.079ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
12022025-01-10T05:16:23.079ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
12032025-01-10T05:16:23.082ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
12042025-01-10T05:16:23.082ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
12052025-01-10T05:16:23.084ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
12062025-01-10T05:16:23.084ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
12072025-01-10T05:16:23.091ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 434.865553ms iteration = 22 status = {}
12082025-01-10T05:16:27.549ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 295 local_addr = [fd00:1122:3344:101::c]:57161 method = GET remote_addr = [fd00:1122:3344:101::d]:46894 req_id = 78fd579b-341a-486a-be98-4751dc7fc025 response_code = 200 uri = /3ca89dd1-7ada-4fe6-8600-a84d41dfa9d8
12092025-01-10T05:16:32.656ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 65 reason = Timeout
12102025-01-10T05:16:32.656ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 65 reason = Timeout
12112025-01-10T05:16:32.656ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12122025-01-10T05:16:32.656ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
12132025-01-10T05:16:32.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
12142025-01-10T05:16:32.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12152025-01-10T05:16:32.659ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12162025-01-10T05:16:32.661ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
12172025-01-10T05:16:32.661ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12182025-01-10T05:16:32.662ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12192025-01-10T05:16:32.664ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12202025-01-10T05:16:32.664ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12212025-01-10T05:16:32.665ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12222025-01-10T05:16:32.668ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 43ea24da-5730-4a85-9a8e-76459f9b69b1, lookup_type: ById(43ea24da-5730-4a85-9a8e-76459f9b69b1) } result = Ok(())
12232025-01-10T05:16:32.668ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12242025-01-10T05:16:32.668ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12252025-01-10T05:16:32.676ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 original_time_created = 2025-01-10 05:05:43.319091 UTC target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 time_created = 2025-01-10 05:05:43.319091 UTC
12262025-01-10T05:16:32.676ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.476405ms iteration = 65 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"43ea24da-5730-4a85-9a8e-76459f9b69b1","time_created":"2025-01-10T05:05:43.319091Z"}
12272025-01-10T05:16:32.686ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
12282025-01-10T05:16:32.686ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
12292025-01-10T05:16:32.713ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 56.878706ms iteration = 65 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
12302025-01-10T05:16:37.549ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 272 local_addr = [fd00:1122:3344:101::c]:57161 method = GET remote_addr = [fd00:1122:3344:101::d]:46894 req_id = d77d112f-688e-44af-82ce-c5fdcb0e995f response_code = 200 uri = /3ca89dd1-7ada-4fe6-8600-a84d41dfa9d8
12312025-01-10T05:16:42.656ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 66 reason = Timeout
12322025-01-10T05:16:42.656ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 66 reason = Timeout
12332025-01-10T05:16:42.656ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12342025-01-10T05:16:42.656ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
12352025-01-10T05:16:42.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12362025-01-10T05:16:42.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
12372025-01-10T05:16:42.658ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12382025-01-10T05:16:42.661ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
12392025-01-10T05:16:42.661ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12402025-01-10T05:16:42.661ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12412025-01-10T05:16:42.663ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12422025-01-10T05:16:42.663ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12432025-01-10T05:16:42.665ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12442025-01-10T05:16:42.667ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 43ea24da-5730-4a85-9a8e-76459f9b69b1, lookup_type: ById(43ea24da-5730-4a85-9a8e-76459f9b69b1) } result = Ok(())
12452025-01-10T05:16:42.667ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12462025-01-10T05:16:42.668ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12472025-01-10T05:16:42.676ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 original_time_created = 2025-01-10 05:05:43.319091 UTC target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 time_created = 2025-01-10 05:05:43.319091 UTC
12482025-01-10T05:16:42.676ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 19.996019ms iteration = 66 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"43ea24da-5730-4a85-9a8e-76459f9b69b1","time_created":"2025-01-10T05:05:43.319091Z"}
12492025-01-10T05:16:42.686ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
12502025-01-10T05:16:42.686ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
12512025-01-10T05:16:42.715ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 58.955253ms iteration = 66 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
12522025-01-10T05:16:47.548ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 295 local_addr = [fd00:1122:3344:101::c]:57161 method = GET remote_addr = [fd00:1122:3344:101::d]:46894 req_id = d6d87be2-f6fb-4263-911d-34e9654a0670 response_code = 200 uri = /3ca89dd1-7ada-4fe6-8600-a84d41dfa9d8
12532025-01-10T05:16:52.541ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::c]:12221 remote_addr = [fd00:1122:3344:101::d]:63450
12542025-01-10T05:16:52.541ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:63450 req_id = 8445262d-a4a0-493e-a642-da2e7749f3eb roles = RoleSet { roles: {} } uri = /metrics/collectors/b6508fc7-5132-4a68-971d-9bf335b84514/producers?sort_by=id_ascending
12552025-01-10T05:16:52.542ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:63450 req_id = 8445262d-a4a0-493e-a642-da2e7749f3eb resource = Database result = Ok(()) uri = /metrics/collectors/b6508fc7-5132-4a68-971d-9bf335b84514/producers?sort_by=id_ascending
12562025-01-10T05:16:52.545ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 3264 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:63450 req_id = 8445262d-a4a0-493e-a642-da2e7749f3eb response_code = 200 uri = /metrics/collectors/b6508fc7-5132-4a68-971d-9bf335b84514/producers?sort_by=id_ascending
12572025-01-10T05:16:52.545ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:63450 req_id = dd046285-ad10-46e1-b6d6-2ada2812812a roles = RoleSet { roles: {} } uri = /metrics/collectors/b6508fc7-5132-4a68-971d-9bf335b84514/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkNTAzY2RhOC01Mjc5LTRhNGMtOTFlNi02NjNjMTcyM2U2NzYifX0%3D
12582025-01-10T05:16:52.545ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:63450 req_id = dd046285-ad10-46e1-b6d6-2ada2812812a resource = Database result = Ok(()) uri = /metrics/collectors/b6508fc7-5132-4a68-971d-9bf335b84514/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkNTAzY2RhOC01Mjc5LTRhNGMtOTFlNi02NjNjMTcyM2U2NzYifX0%3D
12592025-01-10T05:16:52.547ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 2428 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:63450 req_id = dd046285-ad10-46e1-b6d6-2ada2812812a response_code = 200 uri = /metrics/collectors/b6508fc7-5132-4a68-971d-9bf335b84514/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkNTAzY2RhOC01Mjc5LTRhNGMtOTFlNi02NjNjMTcyM2U2NzYifX0%3D
12602025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 23 reason = Timeout
12612025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
12622025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 23 reason = Timeout
12632025-01-10T05:16:52.656ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
12642025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
12652025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 23 reason = Timeout
12662025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 25 reason = Timeout
12672025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
12682025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 14 reason = Timeout
12692025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
12702025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 12 reason = Timeout
12712025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 12 reason = Timeout
12722025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 12 reason = Timeout
12732025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
12742025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
12752025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
12762025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 23 reason = Timeout
12772025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
12782025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 12 reason = Timeout
12792025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
12802025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 12 reason = Timeout
12812025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 23 reason = Timeout
12822025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 23 reason = Timeout
12832025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
12842025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 12 reason = Timeout
12852025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 12 reason = Timeout
12862025-01-10T05:16:52.656ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-01-10 05:06:52.656586956 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
12872025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 23 reason = Timeout
12882025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
12892025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
12902025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
12912025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 67 reason = Timeout
12922025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 23 reason = Timeout
12932025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12942025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
12952025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 23 reason = Timeout
12962025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 23 reason = Timeout
12972025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
12982025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
12992025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 23 reason = Timeout
13002025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
13012025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
13022025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
13032025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 12 reason = Timeout
13042025-01-10T05:16:52.656ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
13052025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 13 reason = Timeout
13062025-01-10T05:16:52.656ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:83 reason = blueprint disabled target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1
13072025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
13082025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 12 reason = Timeout
13092025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 8.48\u{b5}s iteration = 13 status = {"enabled":false,"target_id":"43ea24da-5730-4a85-9a8e-76459f9b69b1"}
13102025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 23 reason = Timeout
13112025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 67 reason = Timeout
13122025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
13132025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
13142025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
13152025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 14 reason = Timeout
13162025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 23 reason = Timeout
13172025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 12 reason = Timeout
13182025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
13192025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
13202025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
13212025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
13222025-01-10T05:16:52.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
13232025-01-10T05:16:52.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
13242025-01-10T05:16:52.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
13252025-01-10T05:16:52.657ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("30a043aa-79b8-40f0-889b-b1aa4686508a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("95b462f2-3c97-4463-9c08-5404518f33d2.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("95b462f2-3c97-4463-9c08-5404518f33d2.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 27023, tv_nsec: 164166652 } })
13262025-01-10T05:16:52.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
13272025-01-10T05:16:52.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
13282025-01-10T05:16:52.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
13292025-01-10T05:16:52.657ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("53514ca5-28a5-4b76-afd8-362873acf1d4.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("b3737e73-ceea-4b83-966e-05b55558955a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("e93887bd-ffe8-46b8-ad6f-92e0665448ab.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("e93887bd-ffe8-46b8-ad6f-92e0665448ab.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 27023, tv_nsec: 164221974 } })
13302025-01-10T05:16:52.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
13312025-01-10T05:16:52.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
13322025-01-10T05:16:52.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
13332025-01-10T05:16:52.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
13342025-01-10T05:16:52.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
13352025-01-10T05:16:52.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
13362025-01-10T05:16:52.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
13372025-01-10T05:16:52.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
13382025-01-10T05:16:52.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
13392025-01-10T05:16:52.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
13402025-01-10T05:16:52.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
13412025-01-10T05:16:52.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
13422025-01-10T05:16:52.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
13432025-01-10T05:16:52.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
13442025-01-10T05:16:52.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
13452025-01-10T05:16:52.657ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
13462025-01-10T05:16:52.657ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 997.623\u{b5}s iteration = 12 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
13472025-01-10T05:16:52.657ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
13482025-01-10T05:16:52.657ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.20117ms iteration = 12 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353"]}
13492025-01-10T05:16:52.657ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 23 reason = Timeout
13502025-01-10T05:16:52.657ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
13512025-01-10T05:16:52.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
13522025-01-10T05:16:52.659ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13532025-01-10T05:16:52.660ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13542025-01-10T05:16:52.660ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.738951ms iteration = 23 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
13552025-01-10T05:16:52.660ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
13562025-01-10T05:16:52.660ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
13572025-01-10T05:16:52.661ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
13582025-01-10T05:16:52.661ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
13592025-01-10T05:16:52.661ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
13602025-01-10T05:16:52.662ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
13612025-01-10T05:16:52.662ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
13622025-01-10T05:16:52.662ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
13632025-01-10T05:16:52.663ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 27023, tv_nsec: 170675183 } })
13642025-01-10T05:16:52.663ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1103
13652025-01-10T05:16:52.665ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 27023, tv_nsec: 172327957 } })
13662025-01-10T05:16:52.665ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1103
13672025-01-10T05:16:52.667ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 11.166702ms iteration = 12 status = {"errors":[],"found_port_ok":[]}
13682025-01-10T05:16:52.667ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-01-10 05:06:52.656586956 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
13692025-01-10T05:16:52.668ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 11.4063ms iteration = 12 status = {"expiration":"2025-01-10T05:06:52.656586956Z","pruned":[]}
13702025-01-10T05:16:52.669ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13712025-01-10T05:16:52.671ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
13722025-01-10T05:16:52.671ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
13732025-01-10T05:16:52.671ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
13742025-01-10T05:16:52.677ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13752025-01-10T05:16:52.677ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13762025-01-10T05:16:52.678ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
13772025-01-10T05:16:52.678ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
13782025-01-10T05:16:52.681ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
13792025-01-10T05:16:52.681ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
13802025-01-10T05:16:52.681ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
13812025-01-10T05:16:52.681ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
13822025-01-10T05:16:52.681ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
13832025-01-10T05:16:52.681ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
13842025-01-10T05:16:52.682ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13852025-01-10T05:16:52.683ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
13862025-01-10T05:16:52.683ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
13872025-01-10T05:16:52.685ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
13882025-01-10T05:16:52.685ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
13892025-01-10T05:16:52.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
13902025-01-10T05:16:52.691ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13912025-01-10T05:16:52.692ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
13922025-01-10T05:16:52.692ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
13932025-01-10T05:16:52.695ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
13942025-01-10T05:16:52.701ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
13952025-01-10T05:16:52.701ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 44.718301ms iteration = 67 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
13962025-01-10T05:16:52.701ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
13972025-01-10T05:16:52.702ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 27023, tv_nsec: 209594876 } })
13982025-01-10T05:16:52.702ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
13992025-01-10T05:16:52.702ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1103 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08
14002025-01-10T05:16:52.703ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
14012025-01-10T05:16:52.703ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
14022025-01-10T05:16:52.704ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14032025-01-10T05:16:52.704ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
14042025-01-10T05:16:52.704ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14052025-01-10T05:16:52.707ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
14062025-01-10T05:16:52.707ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
14072025-01-10T05:16:52.707ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
14082025-01-10T05:16:52.708ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 52.046369ms iteration = 23 status = {"errors":[],"garbage_collect_requested":[]}
14092025-01-10T05:16:52.708ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
14102025-01-10T05:16:52.708ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
14112025-01-10T05:16:52.708ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
14122025-01-10T05:16:52.712ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
14132025-01-10T05:16:52.712ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
14142025-01-10T05:16:52.714ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14152025-01-10T05:16:52.717ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
14162025-01-10T05:16:52.717ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
14172025-01-10T05:16:52.717ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
14182025-01-10T05:16:52.718ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
14192025-01-10T05:16:52.718ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14202025-01-10T05:16:52.718ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
14212025-01-10T05:16:52.719ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
14222025-01-10T05:16:52.719ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
14232025-01-10T05:16:52.720ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 64.455392ms iteration = 23 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
14242025-01-10T05:16:52.722ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
14252025-01-10T05:16:52.722ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
14262025-01-10T05:16:52.722ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
14272025-01-10T05:16:52.724ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
14282025-01-10T05:16:52.725ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
14292025-01-10T05:16:52.727ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
14302025-01-10T05:16:52.727ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:171 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
14312025-01-10T05:16:52.727ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 70.88248ms iteration = 12 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
14322025-01-10T05:16:52.729ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
14332025-01-10T05:16:52.729ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
14342025-01-10T05:16:52.730ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14352025-01-10T05:16:52.731ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
14362025-01-10T05:16:52.731ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
14372025-01-10T05:16:52.732ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
14382025-01-10T05:16:52.732ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
14392025-01-10T05:16:52.732ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
14402025-01-10T05:16:52.732ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14412025-01-10T05:16:52.735ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59, lookup_type: ById(a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59) } result = Ok(())
14422025-01-10T05:16:52.735ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
14432025-01-10T05:16:52.735ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
14442025-01-10T05:16:52.738ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 81.223566ms iteration = 23 status = 0
14452025-01-10T05:16:52.742ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
14462025-01-10T05:16:52.742ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
14472025-01-10T05:16:52.742ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
14482025-01-10T05:16:52.742ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
14492025-01-10T05:16:52.742ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
14502025-01-10T05:16:52.744ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 86.336812ms iteration = 23 status = {"errors":[],"finish_invoked_ok":[]}
14512025-01-10T05:16:52.744ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14522025-01-10T05:16:52.746ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14532025-01-10T05:16:52.747ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
14542025-01-10T05:16:52.747ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
14552025-01-10T05:16:52.747ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
14562025-01-10T05:16:52.748ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
14572025-01-10T05:16:52.748ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14582025-01-10T05:16:52.749ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
14592025-01-10T05:16:52.749ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14602025-01-10T05:16:52.749ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14612025-01-10T05:16:52.749ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
14622025-01-10T05:16:52.749ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
14632025-01-10T05:16:52.749ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
14642025-01-10T05:16:52.751ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
14652025-01-10T05:16:52.754ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
14662025-01-10T05:16:52.754ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
14672025-01-10T05:16:52.762ZINFOnexus (BackgroundTasks): found Silo/DNS/TLS config (changed) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-01-10T05:05:45.360170Z, time_modified: 2025-01-10T05:05:45.360170Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 48a76ac8-2f94-4847-be0f-1cb4bfb661aa, db_silo: Silo { identity: SiloIdentity { id: 48a76ac8-2f94-4847-be0f-1cb4bfb661aa, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-01-10T05:06:09.192249Z, time_modified: 2025-01-10T05:06:09.192249Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "0de51367feb2d6f1959542dbb73fb728546b6d8dd71a0ec2fd5083120a87d20f" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 48a76ac8-2f94-4847-be0f-1cb4bfb661aa, db_silo: Silo { identity: SiloIdentity { id: 48a76ac8-2f94-4847-be0f-1cb4bfb661aa, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-01-10T05:06:09.192249Z, time_modified: 2025-01-10T05:06:09.192249Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "0de51367feb2d6f1959542dbb73fb728546b6d8dd71a0ec2fd5083120a87d20f" }] }) } file = nexus/src/app/background/tasks/external_endpoints.rs:94
14682025-01-10T05:16:52.762ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 106.159746ms iteration = 12 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"48a76ac8-2f94-4847-be0f-1cb4bfb661aa","tls_certs":["0de51367feb2d6f1959542dbb73fb728546b6d8dd71a0ec2fd5083120a87d20f"]}},"default_endpoint":{"silo_id":"48a76ac8-2f94-4847-be0f-1cb4bfb661aa","tls_certs":["0de51367feb2d6f1959542dbb73fb728546b6d8dd71a0ec2fd5083120a87d20f"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
14692025-01-10T05:16:52.768ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 112.298544ms iteration = 23 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
14702025-01-10T05:16:52.769ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
14712025-01-10T05:16:52.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
14722025-01-10T05:16:52.771ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
14732025-01-10T05:16:52.771ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
14742025-01-10T05:16:52.771ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 114.682512ms iteration = 12 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
14752025-01-10T05:16:52.777ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14762025-01-10T05:16:52.781ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 43ea24da-5730-4a85-9a8e-76459f9b69b1, lookup_type: ById(43ea24da-5730-4a85-9a8e-76459f9b69b1) } result = Ok(())
14772025-01-10T05:16:52.781ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
14782025-01-10T05:16:52.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
14792025-01-10T05:16:52.789ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
14802025-01-10T05:16:52.789ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
14812025-01-10T05:16:52.789ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
14822025-01-10T05:16:52.789ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
14832025-01-10T05:16:52.789ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
14842025-01-10T05:16:52.807ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
14852025-01-10T05:16:52.807ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
14862025-01-10T05:16:52.828ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 171.977621ms iteration = 25 status = {"physical_disks_added":0}
14872025-01-10T05:16:52.850ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1111 zone_address = fd00:1122:3344:101::2
14882025-01-10T05:16:52.850ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
14892025-01-10T05:16:52.851ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "d317612f-bd35-478f-9004-20f035f0e286", "content-length": "26", "date": "Fri, 10 Jan 2025 05:16:52 GMT"} })
14902025-01-10T05:16:52.851ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1116 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
14912025-01-10T05:16:52.853ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1148 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
14922025-01-10T05:16:52.898ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1111 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 zone_address = fd00:1122:3344:101::2
14932025-01-10T05:16:52.898ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
14942025-01-10T05:16:52.899ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a483195-d3da-4b62-a157-4b562847f587", "content-length": "26", "date": "Fri, 10 Jan 2025 05:16:52 GMT"} })
14952025-01-10T05:16:52.899ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1116 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
14962025-01-10T05:16:52.900ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1148 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 79fde928-482f-41ab-aee3-3f2478145f08
14972025-01-10T05:16:52.902ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1111 zone_address = fd00:1122:3344:101::2
14982025-01-10T05:16:52.902ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
14992025-01-10T05:16:52.903ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ab8a596-2144-48eb-947b-2602bab99775", "content-length": "26", "date": "Fri, 10 Jan 2025 05:16:52 GMT"} })
15002025-01-10T05:16:52.903ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1116 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
15012025-01-10T05:16:52.905ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1148 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
15022025-01-10T05:16:52.935ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
15032025-01-10T05:16:52.965ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
15042025-01-10T05:16:52.989ZDEBGnexus (BackgroundTasks): client request SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = instance_watcher body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vmms/94045aaa-6f77-4209-bb66-cde829bf3dd0/state
15052025-01-10T05:16:53.011ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 original_time_created = 2025-01-10 05:05:43.319091 UTC target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 time_created = 2025-01-10 05:05:43.319091 UTC
15062025-01-10T05:16:53.011ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 354.845885ms iteration = 67 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"43ea24da-5730-4a85-9a8e-76459f9b69b1","time_created":"2025-01-10T05:05:43.319091Z"}
15072025-01-10T05:16:53.027ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
15082025-01-10T05:16:53.027ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
15092025-01-10T05:16:53.027ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
15102025-01-10T05:16:53.028ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
15112025-01-10T05:16:53.032ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 375.891939ms iteration = 23 status = {}
15122025-01-10T05:16:53.033ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 376.644304ms iteration = 23 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
15132025-01-10T05:16:53.033ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
15142025-01-10T05:16:53.033ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = bca3dac0-9831-415b-86b5-c6c6cc12ed56 dns_zone_name = oxide.test found_so_far = 0 version = 2
15152025-01-10T05:16:53.033ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
15162025-01-10T05:16:53.034ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
15172025-01-10T05:16:53.038ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15182025-01-10T05:16:53.041ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
15192025-01-10T05:16:53.041ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
15202025-01-10T05:16:53.042ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
15212025-01-10T05:16:53.046ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
15222025-01-10T05:16:53.046ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "815e1a86-2ead-4889-bdfe-2a23a2581304", "content-length": "2", "date": "Fri, 10 Jan 2025 05:16:52 GMT"} })
15232025-01-10T05:16:53.048ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
15242025-01-10T05:16:53.048ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
15252025-01-10T05:16:53.051ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 394.673978ms iteration = 23 status = null
15262025-01-10T05:16:53.065ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
15272025-01-10T05:16:53.065ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
15282025-01-10T05:16:53.065ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 408.781627ms iteration = 23 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
15292025-01-10T05:16:53.065ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15302025-01-10T05:16:53.068ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
15312025-01-10T05:16:53.068ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
15322025-01-10T05:16:53.068ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
15332025-01-10T05:16:53.070ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = bca3dac0-9831-415b-86b5-c6c6cc12ed56 dns_zone_name = oxide.test found_so_far = 1 version = 2
15342025-01-10T05:16:53.070ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
15352025-01-10T05:16:53.070ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-01-10 05:06:09.259013 UTC dns_group = external generation = 2
15362025-01-10T05:16:53.070ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 414.063167ms iteration = 12 status = {"generation":2}
15372025-01-10T05:16:53.075ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
15382025-01-10T05:16:53.075ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 70f09b61-dde8-4da2-af2f-06138d5a975e dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
15392025-01-10T05:16:53.075ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
15402025-01-10T05:16:53.075ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
15412025-01-10T05:16:53.103ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15422025-01-10T05:16:53.107ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
15432025-01-10T05:16:53.107ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
15442025-01-10T05:16:53.107ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
15452025-01-10T05:16:53.110ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 70f09b61-dde8-4da2-af2f-06138d5a975e dns_zone_name = control-plane.oxide.internal found_so_far = 51 version = 1
15462025-01-10T05:16:53.110ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
15472025-01-10T05:16:53.110ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-01-10 05:06:08.425967 UTC dns_group = internal generation = 1
15482025-01-10T05:16:53.110ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 454.155919ms iteration = 12 status = {"generation":1}
15492025-01-10T05:16:53.115ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
15502025-01-10T05:16:53.116ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 459.253144ms iteration = 12 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
15512025-01-10T05:16:53.128ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
15522025-01-10T05:16:53.129ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
15532025-01-10T05:16:53.130ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15542025-01-10T05:16:53.132ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
15552025-01-10T05:16:53.132ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
15562025-01-10T05:16:53.133ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
15572025-01-10T05:16:53.134ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15582025-01-10T05:16:53.136ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
15592025-01-10T05:16:53.136ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
15602025-01-10T05:16:53.136ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
15612025-01-10T05:16:53.183ZINFOnexus (BackgroundTasks): VPC route manager sled a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
15622025-01-10T05:16:53.183ZDEBGnexus (BackgroundTasks): client request SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
15632025-01-10T05:16:53.184ZDEBGnexus (BackgroundTasks): client response SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "15b0a388-acec-415b-8b54-2aff53acb15b", "content-length": "583", "date": "Fri, 10 Jan 2025 05:16:52 GMT"} })
15642025-01-10T05:16:53.184ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
15652025-01-10T05:16:53.184ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
15662025-01-10T05:16:53.187ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
15672025-01-10T05:16:53.188ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "4a59dcc1-618c-452d-985a-b098a23313b9", "date": "Fri, 10 Jan 2025 05:16:52 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
15682025-01-10T05:16:53.188ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
15692025-01-10T05:16:53.189ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
15702025-01-10T05:16:53.190ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "f9e33056-6e3c-4a1f-9f6a-95c34cb6f694", "date": "Fri, 10 Jan 2025 05:16:52 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
15712025-01-10T05:16:53.190ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
15722025-01-10T05:16:53.192ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
15732025-01-10T05:16:53.192ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 536.14458ms iteration = 14 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
15742025-01-10T05:16:53.246ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {ba175292-1bb0-46be-b0eb-886eb79f1ba3: {10.151.2.176: {6703c9a8-9630-4196-98b7-6fd428c2a9e7}, 10.151.2.175: {6703c9a8-9630-4196-98b7-6fd428c2a9e7}}} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
15752025-01-10T05:16:53.246ZDEBGnexus (BackgroundTasks): client request SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
15762025-01-10T05:16:53.247ZDEBGnexus (BackgroundTasks): client response SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "2baa65a5-12a0-4ac5-932e-1fbb92b294c2", "date": "Fri, 10 Jan 2025 05:16:52 GMT"} })
15772025-01-10T05:16:53.247ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
15782025-01-10T05:16:53.247ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
15792025-01-10T05:16:53.250ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
15802025-01-10T05:16:53.250ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
15812025-01-10T05:16:53.253ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
15822025-01-10T05:16:53.253ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
15832025-01-10T05:16:53.292ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
15842025-01-10T05:16:53.293ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "3a98bd95-ee93-402a-9226-f80dbeb7c326", "date": "Fri, 10 Jan 2025 05:16:53 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
15852025-01-10T05:16:53.293ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
15862025-01-10T05:16:53.295ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "c274eab4-e420-4997-8f26-1bd251628bae", "date": "Fri, 10 Jan 2025 05:16:52 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
15872025-01-10T05:16:53.295ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
15882025-01-10T05:16:53.296ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "61ffe3a4-372e-439a-942e-00ce3727f9cd", "date": "Fri, 10 Jan 2025 05:16:53 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
15892025-01-10T05:16:53.296ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
15902025-01-10T05:16:53.298ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
15912025-01-10T05:16:53.298ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 641.825539ms iteration = 14 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
15922025-01-10T05:16:53.318ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
15932025-01-10T05:16:53.318ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
15942025-01-10T05:16:53.322ZDEBGnexus (BackgroundTasks): client request SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
15952025-01-10T05:16:53.323ZDEBGnexus (BackgroundTasks): client response SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c814f5e-555d-48f9-a724-b41155c8fa4c", "content-length": "810", "date": "Fri, 10 Jan 2025 05:16:52 GMT"} })
15962025-01-10T05:16:53.323ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {VirtualNetworkInterfaceHost { physical_host_ip: fd00:1122:3344:101::1, virtual_ip: 172.30.0.5, virtual_mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), vni: Vni(6445912) }} sled = sock
15972025-01-10T05:16:53.323ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
15982025-01-10T05:16:53.323ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
15992025-01-10T05:16:53.324ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 668.293788ms iteration = 23 status = {}
16002025-01-10T05:16:53.325ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 5263359f-a80d-45e0-9595-0b15b904e78f, rack_id: 79fde928-482f-41ab-aee3-3f2478145f08, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(77858eb2-1069-4750-be5e-f3dafe20063a) } rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 switch_location = Switch0
16012025-01-10T05:16:53.325ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
16022025-01-10T05:16:53.325ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
16032025-01-10T05:16:53.325ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
16042025-01-10T05:16:53.325ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
16052025-01-10T05:16:53.328ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
16062025-01-10T05:16:53.328ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
16072025-01-10T05:16:53.330ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
16082025-01-10T05:16:53.330ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
16092025-01-10T05:16:53.333ZINFOnexus (BackgroundTasks): VPC route manager sled a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
16102025-01-10T05:16:53.333ZINFOnexus (BackgroundTasks): VPC route manager sled a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
16112025-01-10T05:16:53.333ZDEBGnexus (BackgroundTasks): client request SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
16122025-01-10T05:16:53.333ZDEBGnexus (BackgroundTasks): client response SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "fee03e2d-fbdb-4215-8b86-de6b6444f2e5", "date": "Fri, 10 Jan 2025 05:16:52 GMT"} })
16132025-01-10T05:16:53.335ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 679.071668ms iteration = 23 status = {}
16142025-01-10T05:16:53.335ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
16152025-01-10T05:16:53.336ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed30354e-76f7-44dc-8374-e79dd6feaeb8", "content-length": "134", "date": "Fri, 10 Jan 2025 05:16:52 GMT"} })
16162025-01-10T05:16:53.336ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
16172025-01-10T05:16:53.336ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 77858eb2-1069-4750-be5e-f3dafe20063a
16182025-01-10T05:16:53.336ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
16192025-01-10T05:16:53.336ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
16202025-01-10T05:16:53.336ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
16212025-01-10T05:16:53.336ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
16222025-01-10T05:16:53.336ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
16232025-01-10T05:16:53.336ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
16242025-01-10T05:16:53.336ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
16252025-01-10T05:16:53.336ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
16262025-01-10T05:16:53.337ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08
16272025-01-10T05:16:53.337ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
16282025-01-10T05:16:53.337ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
16292025-01-10T05:16:53.339ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
16302025-01-10T05:16:53.339ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c5aa4f5-f5e0-4814-a328-602e7e6cc9f9", "content-length": "2", "date": "Fri, 10 Jan 2025 05:16:52 GMT"} })
16312025-01-10T05:16:53.339ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
16322025-01-10T05:16:53.339ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "eab7d24a-dc54-45fb-9602-40f3376e1195", "content-length": "52", "date": "Fri, 10 Jan 2025 05:16:52 GMT"} })
16332025-01-10T05:16:53.339ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 switch_location = Switch0
16342025-01-10T05:16:53.339ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
16352025-01-10T05:16:53.438ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "409d2a5a-e229-4106-847b-684fc2655e14", "date": "Fri, 10 Jan 2025 05:16:52 GMT"} })
16362025-01-10T05:16:53.438ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 switch = Switch0
16372025-01-10T05:16:53.438ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
16382025-01-10T05:16:53.438ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "3edced8b-dc3f-44c7-b5c1-f68af3eec60d", "content-length": "682", "date": "Fri, 10 Jan 2025 05:16:52 GMT"} })
16392025-01-10T05:16:53.438ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
16402025-01-10T05:16:53.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
16412025-01-10T05:16:53.443ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
16422025-01-10T05:16:53.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
16432025-01-10T05:16:53.447ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
16442025-01-10T05:16:53.447ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
16452025-01-10T05:16:53.449ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
16462025-01-10T05:16:53.450ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
16472025-01-10T05:16:53.457ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 800.403866ms iteration = 23 status = {}
16482025-01-10T05:16:53.539ZDEBGnexus (BackgroundTasks): client response SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = instance_watcher result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vmms/94045aaa-6f77-4209-bb66-cde829bf3dd0/state", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d965a09-92d0-4a05-83a3-124097a3d3a5", "content-length": "132", "date": "Fri, 10 Jan 2025 05:16:52 GMT"} })
16492025-01-10T05:16:53.544ZDEBGnexus (BackgroundTasks): updating instance state background_task = instance_watcher instance_id = 7bf94b83-d698-4a23-bd13-e9fb4af61dd4 sled_id = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 state = SledVmmState { vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(8), time_updated: 2025-01-10T05:16:53.479318696Z }, migration_in: None, migration_out: None } vmm_id = 94045aaa-6f77-4209-bb66-cde829bf3dd0
16502025-01-10T05:16:53.544ZINFOnexus (BackgroundTasks): received new VMM runtime state from sled agent background_task = instance_watcher file = nexus/src/app/instance.rs:2077 instance_id = 7bf94b83-d698-4a23-bd13-e9fb4af61dd4 migration_state = Migrations { migration_in: None, migration_out: None } propolis_id = 94045aaa-6f77-4209-bb66-cde829bf3dd0 sled_id = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 vmm_id = 94045aaa-6f77-4209-bb66-cde829bf3dd0 vmm_state = VmmRuntimeState { state: Destroyed, gen: Generation(8), time_updated: 2025-01-10T05:16:53.479318696Z }
16512025-01-10T05:16:53.544ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher instance_id = 7bf94b83-d698-4a23-bd13-e9fb4af61dd4 roles = RoleSet { roles: {} } sled_id = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 vmm_id = 94045aaa-6f77-4209-bb66-cde829bf3dd0
16522025-01-10T05:16:53.545ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher instance_id = 7bf94b83-d698-4a23-bd13-e9fb4af61dd4 resource = Database result = Ok(()) sled_id = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 vmm_id = 94045aaa-6f77-4209-bb66-cde829bf3dd0
16532025-01-10T05:16:53.556ZDEBGnexus (producer-registration-task): registering / renewing oximeter producer lease with Nexus
16542025-01-10T05:16:53.556ZDEBGnexus (producer-registration-task): using nexus address for registration addr = [fd00:1122:3344:101::c]:12221
16552025-01-10T05:16:53.561ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 1 total_failed = 0 total_incomplete = 0 total_instances = 1 update_sagas_queued = 0
16562025-01-10T05:16:53.563ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 907.265333ms iteration = 23 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{"destroyed":1},"pruned_instances":0,"total_instances":1,"update_sagas_queued":0}
16572025-01-10T05:16:53.593ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::c]:12221/metrics/producers
16582025-01-10T05:16:53.593ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::c]:12221 remote_addr = [fd00:1122:3344:101::c]:33998
16592025-01-10T05:16:53.593ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::c]:33998 req_id = c876525f-bf75-4a40-b240-4e29f075508e roles = RoleSet { roles: {} } uri = /metrics/producers
16602025-01-10T05:16:53.593ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::c]:33998 req_id = c876525f-bf75-4a40-b240-4e29f075508e resource = Database result = Ok(()) uri = /metrics/producers
16612025-01-10T05:16:53.640ZINFOnexus: assigned collector to new producer collector_id = b6508fc7-5132-4a68-971d-9bf335b84514 file = nexus/src/app/oximeter.rs:76 producer_id = 3ca89dd1-7ada-4fe6-8600-a84d41dfa9d8
16622025-01-10T05:16:53.640ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 46641 local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::c]:33998 req_id = c876525f-bf75-4a40-b240-4e29f075508e response_code = 201 uri = /metrics/producers
16632025-01-10T05:16:53.640ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "c876525f-bf75-4a40-b240-4e29f075508e", "content-length": "41", "date": "Fri, 10 Jan 2025 05:16:53 GMT"} })
16642025-01-10T05:16:53.642ZDEBGnexus (producer-registration-task): registered with nexus successfully
16652025-01-10T05:16:53.642ZDEBGnexus (producer-registration-task): pausing until time to renew lease lease_duration = 600s wait_period = 150s
16662025-01-10T05:16:57.550ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 2587 local_addr = [fd00:1122:3344:101::c]:57161 method = GET remote_addr = [fd00:1122:3344:101::d]:46894 req_id = f1938016-9c34-4389-aec0-3f45c02e3239 response_code = 200 uri = /3ca89dd1-7ada-4fe6-8600-a84d41dfa9d8
16672025-01-10T05:17:02.655ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 68 reason = Timeout
16682025-01-10T05:17:02.655ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
16692025-01-10T05:17:02.655ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 68 reason = Timeout
16702025-01-10T05:17:02.655ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16712025-01-10T05:17:02.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
16722025-01-10T05:17:02.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
16732025-01-10T05:17:02.658ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16742025-01-10T05:17:02.661ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
16752025-01-10T05:17:02.661ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16762025-01-10T05:17:02.661ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
16772025-01-10T05:17:02.663ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16782025-01-10T05:17:02.663ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
16792025-01-10T05:17:02.665ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16802025-01-10T05:17:02.668ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 43ea24da-5730-4a85-9a8e-76459f9b69b1, lookup_type: ById(43ea24da-5730-4a85-9a8e-76459f9b69b1) } result = Ok(())
16812025-01-10T05:17:02.668ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16822025-01-10T05:17:02.668ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
16832025-01-10T05:17:02.680ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 original_time_created = 2025-01-10 05:05:43.319091 UTC target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 time_created = 2025-01-10 05:05:43.319091 UTC
16842025-01-10T05:17:02.680ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 25.223819ms iteration = 68 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"43ea24da-5730-4a85-9a8e-76459f9b69b1","time_created":"2025-01-10T05:05:43.319091Z"}
16852025-01-10T05:17:02.687ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
16862025-01-10T05:17:02.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
16872025-01-10T05:17:02.716ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 60.744291ms iteration = 68 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
16882025-01-10T05:17:07.541ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:63450 req_id = aebd6d79-5387-4050-815b-18827e65ba9a roles = RoleSet { roles: {} } uri = /metrics/collectors/b6508fc7-5132-4a68-971d-9bf335b84514/producers?sort_by=id_ascending
16892025-01-10T05:17:07.541ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:63450 req_id = aebd6d79-5387-4050-815b-18827e65ba9a resource = Database result = Ok(()) uri = /metrics/collectors/b6508fc7-5132-4a68-971d-9bf335b84514/producers?sort_by=id_ascending
16902025-01-10T05:17:07.544ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 3184 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:63450 req_id = aebd6d79-5387-4050-815b-18827e65ba9a response_code = 200 uri = /metrics/collectors/b6508fc7-5132-4a68-971d-9bf335b84514/producers?sort_by=id_ascending
16912025-01-10T05:17:07.544ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:63450 req_id = 3bf0113e-e2ca-413c-9f4b-3b6bbe5a02e0 roles = RoleSet { roles: {} } uri = /metrics/collectors/b6508fc7-5132-4a68-971d-9bf335b84514/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkNTAzY2RhOC01Mjc5LTRhNGMtOTFlNi02NjNjMTcyM2U2NzYifX0%3D
16922025-01-10T05:17:07.544ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:63450 req_id = 3bf0113e-e2ca-413c-9f4b-3b6bbe5a02e0 resource = Database result = Ok(()) uri = /metrics/collectors/b6508fc7-5132-4a68-971d-9bf335b84514/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkNTAzY2RhOC01Mjc5LTRhNGMtOTFlNi02NjNjMTcyM2U2NzYifX0%3D
16932025-01-10T05:17:07.546ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 2375 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:63450 req_id = 3bf0113e-e2ca-413c-9f4b-3b6bbe5a02e0 response_code = 200 uri = /metrics/collectors/b6508fc7-5132-4a68-971d-9bf335b84514/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkNTAzY2RhOC01Mjc5LTRhNGMtOTFlNi02NjNjMTcyM2U2NzYifX0%3D
16942025-01-10T05:17:07.548ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 318 local_addr = [fd00:1122:3344:101::c]:57161 method = GET remote_addr = [fd00:1122:3344:101::d]:46894 req_id = 3d410708-1651-4654-b713-b330e2acc00e response_code = 200 uri = /3ca89dd1-7ada-4fe6-8600-a84d41dfa9d8
16952025-01-10T05:17:12.655ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 69 reason = Timeout
16962025-01-10T05:17:12.655ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 69 reason = Timeout
16972025-01-10T05:17:12.655ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16982025-01-10T05:17:12.655ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
16992025-01-10T05:17:12.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17002025-01-10T05:17:12.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
17012025-01-10T05:17:12.657ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17022025-01-10T05:17:12.660ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
17032025-01-10T05:17:12.660ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
17042025-01-10T05:17:12.660ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17052025-01-10T05:17:12.663ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
17062025-01-10T05:17:12.663ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17072025-01-10T05:17:12.665ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17082025-01-10T05:17:12.668ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 43ea24da-5730-4a85-9a8e-76459f9b69b1, lookup_type: ById(43ea24da-5730-4a85-9a8e-76459f9b69b1) } result = Ok(())
17092025-01-10T05:17:12.668ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
17102025-01-10T05:17:12.668ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17112025-01-10T05:17:12.680ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 original_time_created = 2025-01-10 05:05:43.319091 UTC target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 time_created = 2025-01-10 05:05:43.319091 UTC
17122025-01-10T05:17:12.680ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 25.438746ms iteration = 69 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"43ea24da-5730-4a85-9a8e-76459f9b69b1","time_created":"2025-01-10T05:05:43.319091Z"}
17132025-01-10T05:17:12.685ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
17142025-01-10T05:17:12.685ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
17152025-01-10T05:17:12.714ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 58.890951ms iteration = 69 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
17162025-01-10T05:17:17.548ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 342 local_addr = [fd00:1122:3344:101::c]:57161 method = GET remote_addr = [fd00:1122:3344:101::d]:46894 req_id = 5d4bdc9e-fc0e-4aac-a373-e482474ac088 response_code = 200 uri = /3ca89dd1-7ada-4fe6-8600-a84d41dfa9d8
17172025-01-10T05:17:22.655ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 70 reason = Timeout
17182025-01-10T05:17:22.655ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 70 reason = Timeout
17192025-01-10T05:17:22.656ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
17202025-01-10T05:17:22.656ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
17212025-01-10T05:17:22.656ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 24 reason = Timeout
17222025-01-10T05:17:22.656ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 24 reason = Timeout
17232025-01-10T05:17:22.656ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
17242025-01-10T05:17:22.656ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
17252025-01-10T05:17:22.656ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 24 reason = Timeout
17262025-01-10T05:17:22.656ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
17272025-01-10T05:17:22.656ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 26 reason = Timeout
17282025-01-10T05:17:22.656ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 24 reason = Timeout
17292025-01-10T05:17:22.656ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 24 reason = Timeout
17302025-01-10T05:17:22.656ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
17312025-01-10T05:17:22.656ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
17322025-01-10T05:17:22.656ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
17332025-01-10T05:17:22.656ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 24 reason = Timeout
17342025-01-10T05:17:22.656ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
17352025-01-10T05:17:22.656ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 24 reason = Timeout
17362025-01-10T05:17:22.656ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
17372025-01-10T05:17:22.656ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 24 reason = Timeout
17382025-01-10T05:17:22.656ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
17392025-01-10T05:17:22.656ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
17402025-01-10T05:17:22.656ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 24 reason = Timeout
17412025-01-10T05:17:22.656ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 24 reason = Timeout
17422025-01-10T05:17:22.656ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
17432025-01-10T05:17:22.656ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 24 reason = Timeout
17442025-01-10T05:17:22.656ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
17452025-01-10T05:17:22.656ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
17462025-01-10T05:17:22.656ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 24 reason = Timeout
17472025-01-10T05:17:22.656ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 24 reason = Timeout
17482025-01-10T05:17:22.656ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
17492025-01-10T05:17:22.656ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
17502025-01-10T05:17:22.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17512025-01-10T05:17:22.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
17522025-01-10T05:17:22.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
17532025-01-10T05:17:22.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
17542025-01-10T05:17:22.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
17552025-01-10T05:17:22.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
17562025-01-10T05:17:22.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
17572025-01-10T05:17:22.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
17582025-01-10T05:17:22.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
17592025-01-10T05:17:22.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
17602025-01-10T05:17:22.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
17612025-01-10T05:17:22.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
17622025-01-10T05:17:22.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
17632025-01-10T05:17:22.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
17642025-01-10T05:17:22.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
17652025-01-10T05:17:22.658ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 24 reason = Timeout
17662025-01-10T05:17:22.658ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
17672025-01-10T05:17:22.658ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
17682025-01-10T05:17:22.658ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17692025-01-10T05:17:22.659ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.507744ms iteration = 24 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
17702025-01-10T05:17:22.659ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 27053, tv_nsec: 167142089 } })
17712025-01-10T05:17:22.659ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
17722025-01-10T05:17:22.660ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1103
17732025-01-10T05:17:22.660ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
17742025-01-10T05:17:22.660ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 27053, tv_nsec: 168413090 } })
17752025-01-10T05:17:22.661ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1103
17762025-01-10T05:17:22.661ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
17772025-01-10T05:17:22.661ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
17782025-01-10T05:17:22.661ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17792025-01-10T05:17:22.664ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17802025-01-10T05:17:22.665ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
17812025-01-10T05:17:22.665ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
17822025-01-10T05:17:22.665ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
17832025-01-10T05:17:22.665ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
17842025-01-10T05:17:22.666ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17852025-01-10T05:17:22.667ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
17862025-01-10T05:17:22.667ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
17872025-01-10T05:17:22.667ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
17882025-01-10T05:17:22.669ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
17892025-01-10T05:17:22.669ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
17902025-01-10T05:17:22.669ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
17912025-01-10T05:17:22.672ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
17922025-01-10T05:17:22.672ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17932025-01-10T05:17:22.672ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
17942025-01-10T05:17:22.673ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
17952025-01-10T05:17:22.678ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 1 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
17962025-01-10T05:17:22.678ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 22.124838ms iteration = 24 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":1,"total_instances":0,"update_sagas_queued":0}
17972025-01-10T05:17:22.692ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17982025-01-10T05:17:22.692ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
17992025-01-10T05:17:22.692ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
18002025-01-10T05:17:22.692ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 36.608538ms iteration = 24 status = {"errors":[],"garbage_collect_requested":[]}
18012025-01-10T05:17:22.692ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
18022025-01-10T05:17:22.693ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
18032025-01-10T05:17:22.694ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18042025-01-10T05:17:22.695ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
18052025-01-10T05:17:22.695ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
18062025-01-10T05:17:22.695ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
18072025-01-10T05:17:22.695ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
18082025-01-10T05:17:22.695ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 37.334031ms iteration = 24 status = {"errors":[],"finish_invoked_ok":[]}
18092025-01-10T05:17:22.695ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
18102025-01-10T05:17:22.697ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
18112025-01-10T05:17:22.697ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
18122025-01-10T05:17:22.697ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
18132025-01-10T05:17:22.698ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
18142025-01-10T05:17:22.698ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
18152025-01-10T05:17:22.699ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18162025-01-10T05:17:22.700ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
18172025-01-10T05:17:22.700ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
18182025-01-10T05:17:22.701ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
18192025-01-10T05:17:22.701ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
18202025-01-10T05:17:22.701ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
18212025-01-10T05:17:22.702ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18222025-01-10T05:17:22.704ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
18232025-01-10T05:17:22.704ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
18242025-01-10T05:17:22.704ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
18252025-01-10T05:17:22.705ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 27053, tv_nsec: 212864962 } })
18262025-01-10T05:17:22.705ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1103 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08
18272025-01-10T05:17:22.706ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18282025-01-10T05:17:22.708ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 52.181823ms iteration = 70 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
18292025-01-10T05:17:22.708ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
18302025-01-10T05:17:22.708ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
18312025-01-10T05:17:22.708ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
18322025-01-10T05:17:22.710ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18332025-01-10T05:17:22.713ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 43ea24da-5730-4a85-9a8e-76459f9b69b1, lookup_type: ById(43ea24da-5730-4a85-9a8e-76459f9b69b1) } result = Ok(())
18342025-01-10T05:17:22.713ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18352025-01-10T05:17:22.713ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
18362025-01-10T05:17:22.723ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
18372025-01-10T05:17:22.723ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
18382025-01-10T05:17:22.728ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
18392025-01-10T05:17:22.728ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
18402025-01-10T05:17:22.728ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
18412025-01-10T05:17:22.728ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
18422025-01-10T05:17:22.728ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 original_time_created = 2025-01-10 05:05:43.319091 UTC target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 time_created = 2025-01-10 05:05:43.319091 UTC
18432025-01-10T05:17:22.728ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 73.02798ms iteration = 70 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"43ea24da-5730-4a85-9a8e-76459f9b69b1","time_created":"2025-01-10T05:05:43.319091Z"}
18442025-01-10T05:17:22.731ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
18452025-01-10T05:17:22.731ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
18462025-01-10T05:17:22.737ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1111 zone_address = fd00:1122:3344:101::2
18472025-01-10T05:17:22.737ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
18482025-01-10T05:17:22.737ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "2518afb8-aeb1-4133-809a-3c14e204bb4a", "content-length": "26", "date": "Fri, 10 Jan 2025 05:17:22 GMT"} })
18492025-01-10T05:17:22.737ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1116 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
18502025-01-10T05:17:22.739ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1148 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
18512025-01-10T05:17:22.741ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18522025-01-10T05:17:22.741ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 85.583817ms iteration = 24 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
18532025-01-10T05:17:22.743ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1111 zone_address = fd00:1122:3344:101::2
18542025-01-10T05:17:22.743ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
18552025-01-10T05:17:22.743ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "17eeba93-bc42-492f-a7da-934c9b312c2b", "content-length": "26", "date": "Fri, 10 Jan 2025 05:17:22 GMT"} })
18562025-01-10T05:17:22.743ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1116 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
18572025-01-10T05:17:22.744ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59, lookup_type: ById(a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59) } result = Ok(())
18582025-01-10T05:17:22.744ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
18592025-01-10T05:17:22.744ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
18602025-01-10T05:17:22.745ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1148 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
18612025-01-10T05:17:22.746ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 90.782677ms iteration = 24 status = 0
18622025-01-10T05:17:22.753ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 97.217455ms iteration = 26 status = {"physical_disks_added":0}
18632025-01-10T05:17:22.757ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 101.790944ms iteration = 24 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
18642025-01-10T05:17:22.759ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 103.890682ms iteration = 24 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
18652025-01-10T05:17:22.764ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
18662025-01-10T05:17:22.764ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
18672025-01-10T05:17:22.764ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 108.199182ms iteration = 24 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
18682025-01-10T05:17:22.782ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
18692025-01-10T05:17:22.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
18702025-01-10T05:17:22.813ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1111 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 zone_address = fd00:1122:3344:101::2
18712025-01-10T05:17:22.813ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
18722025-01-10T05:17:22.813ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "3cf92446-f8e2-41d5-bfec-f319a52203d9", "content-length": "26", "date": "Fri, 10 Jan 2025 05:17:22 GMT"} })
18732025-01-10T05:17:22.813ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1116 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
18742025-01-10T05:17:22.815ZINFOnexus (BackgroundTasks): VPC route manager sled a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
18752025-01-10T05:17:22.815ZDEBGnexus (BackgroundTasks): client request SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
18762025-01-10T05:17:22.815ZDEBGnexus (BackgroundTasks): client response SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "840bf61e-1ea1-44dd-8847-82c6149bf43f", "content-length": "372", "date": "Fri, 10 Jan 2025 05:17:22 GMT"} })
18772025-01-10T05:17:22.815ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1148 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 79fde928-482f-41ab-aee3-3f2478145f08
18782025-01-10T05:17:22.815ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
18792025-01-10T05:17:22.815ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
18802025-01-10T05:17:22.818ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
18812025-01-10T05:17:22.818ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
18822025-01-10T05:17:22.861ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 205.555291ms iteration = 24 status = {}
18832025-01-10T05:17:22.861ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18842025-01-10T05:17:22.863ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
18852025-01-10T05:17:22.863ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
18862025-01-10T05:17:22.863ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
18872025-01-10T05:17:22.865ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18882025-01-10T05:17:22.867ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
18892025-01-10T05:17:22.867ZDEBGnexus (BackgroundTasks): client request SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
18902025-01-10T05:17:22.867ZDEBGnexus (BackgroundTasks): client response SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "d73046ff-4711-40c2-988a-e21b810f06f2", "date": "Fri, 10 Jan 2025 05:17:22 GMT"} })
18912025-01-10T05:17:22.867ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
18922025-01-10T05:17:22.868ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
18932025-01-10T05:17:22.868ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
18942025-01-10T05:17:22.868ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
18952025-01-10T05:17:22.868ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
18962025-01-10T05:17:22.870ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
18972025-01-10T05:17:22.872ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6ce861c-ced4-43d6-8789-33c8ee7ef30f", "content-length": "2", "date": "Fri, 10 Jan 2025 05:17:22 GMT"} })
18982025-01-10T05:17:22.873ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
18992025-01-10T05:17:22.873ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
19002025-01-10T05:17:22.874ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
19012025-01-10T05:17:22.875ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
19022025-01-10T05:17:22.876ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 221.017793ms iteration = 24 status = null
19032025-01-10T05:17:22.877ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
19042025-01-10T05:17:22.877ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
19052025-01-10T05:17:22.880ZINFOnexus (BackgroundTasks): VPC route manager sled a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
19062025-01-10T05:17:22.880ZDEBGnexus (BackgroundTasks): client request SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
19072025-01-10T05:17:22.881ZDEBGnexus (BackgroundTasks): client response SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "3138a8d7-ad56-45e4-bfbf-07b23723e552", "date": "Fri, 10 Jan 2025 05:17:22 GMT"} })
19082025-01-10T05:17:22.883ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 227.380399ms iteration = 24 status = {}
19092025-01-10T05:17:23.009ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
19102025-01-10T05:17:23.009ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
19112025-01-10T05:17:23.013ZDEBGnexus (BackgroundTasks): client request SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
19122025-01-10T05:17:23.014ZDEBGnexus (BackgroundTasks): client response SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "4203422c-686d-4f33-abac-d73293f29ad0", "content-length": "691", "date": "Fri, 10 Jan 2025 05:17:22 GMT"} })
19132025-01-10T05:17:23.014ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
19142025-01-10T05:17:23.014ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
19152025-01-10T05:17:23.014ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
19162025-01-10T05:17:23.016ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 5263359f-a80d-45e0-9595-0b15b904e78f, rack_id: 79fde928-482f-41ab-aee3-3f2478145f08, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(77858eb2-1069-4750-be5e-f3dafe20063a) } rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 switch_location = Switch0
19172025-01-10T05:17:23.016ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
19182025-01-10T05:17:23.016ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
19192025-01-10T05:17:23.016ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 360.485519ms iteration = 24 status = {}
19202025-01-10T05:17:23.031ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
19212025-01-10T05:17:23.031ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "7cbb19da-71f2-49d0-9dfa-af0751e1b5ef", "content-length": "134", "date": "Fri, 10 Jan 2025 05:17:22 GMT"} })
19222025-01-10T05:17:23.031ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
19232025-01-10T05:17:23.031ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 77858eb2-1069-4750-be5e-f3dafe20063a
19242025-01-10T05:17:23.031ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
19252025-01-10T05:17:23.031ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
19262025-01-10T05:17:23.031ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
19272025-01-10T05:17:23.031ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
19282025-01-10T05:17:23.031ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
19292025-01-10T05:17:23.031ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
19302025-01-10T05:17:23.032ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
19312025-01-10T05:17:23.032ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
19322025-01-10T05:17:23.032ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08
19332025-01-10T05:17:23.032ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
19342025-01-10T05:17:23.032ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
19352025-01-10T05:17:23.034ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
19362025-01-10T05:17:23.034ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "53f2f882-4a41-41be-9232-410764cb9e3a", "content-length": "2", "date": "Fri, 10 Jan 2025 05:17:22 GMT"} })
19372025-01-10T05:17:23.034ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
19382025-01-10T05:17:23.034ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "57a67a55-2f30-49b4-9c97-fe2e977fdf98", "content-length": "52", "date": "Fri, 10 Jan 2025 05:17:22 GMT"} })
19392025-01-10T05:17:23.034ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 switch_location = Switch0
19402025-01-10T05:17:23.034ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
19412025-01-10T05:17:23.130ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "e3b1a06d-85af-422d-8471-e369ff8bb2c7", "date": "Fri, 10 Jan 2025 05:17:22 GMT"} })
19422025-01-10T05:17:23.130ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 switch = Switch0
19432025-01-10T05:17:23.130ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
19442025-01-10T05:17:23.131ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "641a5aa8-c5c7-4e1a-993a-0635fd4ec5dd", "content-length": "682", "date": "Fri, 10 Jan 2025 05:17:23 GMT"} })
19452025-01-10T05:17:23.131ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
19462025-01-10T05:17:23.131ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
19472025-01-10T05:17:23.134ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
19482025-01-10T05:17:23.134ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
19492025-01-10T05:17:23.137ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
19502025-01-10T05:17:23.137ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
19512025-01-10T05:17:23.139ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
19522025-01-10T05:17:23.139ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
19532025-01-10T05:17:23.146ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 490.491928ms iteration = 24 status = {}
19542025-01-10T05:17:27.548ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 296 local_addr = [fd00:1122:3344:101::c]:57161 method = GET remote_addr = [fd00:1122:3344:101::d]:46894 req_id = 3f39b189-40fc-4335-b98c-366432b614c5 response_code = 200 uri = /3ca89dd1-7ada-4fe6-8600-a84d41dfa9d8
19552025-01-10T05:17:32.655ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 71 reason = Timeout
19562025-01-10T05:17:32.655ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 71 reason = Timeout
19572025-01-10T05:17:32.655ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
19582025-01-10T05:17:32.655ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
19592025-01-10T05:17:32.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
19602025-01-10T05:17:32.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
19612025-01-10T05:17:32.657ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19622025-01-10T05:17:32.658ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
19632025-01-10T05:17:32.658ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
19642025-01-10T05:17:32.660ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.996222ms iteration = 71 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
19652025-01-10T05:17:32.660ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
19662025-01-10T05:17:32.660ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
19672025-01-10T05:17:32.660ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
19682025-01-10T05:17:32.663ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
19692025-01-10T05:17:32.663ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
19702025-01-10T05:17:32.664ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19712025-01-10T05:17:32.667ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 43ea24da-5730-4a85-9a8e-76459f9b69b1, lookup_type: ById(43ea24da-5730-4a85-9a8e-76459f9b69b1) } result = Ok(())
19722025-01-10T05:17:32.667ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
19732025-01-10T05:17:32.667ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
19742025-01-10T05:17:32.679ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 original_time_created = 2025-01-10 05:05:43.319091 UTC target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 time_created = 2025-01-10 05:05:43.319091 UTC
19752025-01-10T05:17:32.679ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 23.871565ms iteration = 71 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"43ea24da-5730-4a85-9a8e-76459f9b69b1","time_created":"2025-01-10T05:05:43.319091Z"}
19762025-01-10T05:17:37.540ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:63450 req_id = f6a12d37-53bc-43e3-bf75-25632e247da1 roles = RoleSet { roles: {} } uri = /metrics/collectors/b6508fc7-5132-4a68-971d-9bf335b84514/producers?sort_by=id_ascending
19772025-01-10T05:17:37.541ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:63450 req_id = f6a12d37-53bc-43e3-bf75-25632e247da1 resource = Database result = Ok(()) uri = /metrics/collectors/b6508fc7-5132-4a68-971d-9bf335b84514/producers?sort_by=id_ascending
19782025-01-10T05:17:37.543ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 3176 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:63450 req_id = f6a12d37-53bc-43e3-bf75-25632e247da1 response_code = 200 uri = /metrics/collectors/b6508fc7-5132-4a68-971d-9bf335b84514/producers?sort_by=id_ascending
19792025-01-10T05:17:37.544ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:63450 req_id = 94c11070-733f-45a2-ac1b-ce895d701e8f roles = RoleSet { roles: {} } uri = /metrics/collectors/b6508fc7-5132-4a68-971d-9bf335b84514/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkNTAzY2RhOC01Mjc5LTRhNGMtOTFlNi02NjNjMTcyM2U2NzYifX0%3D
19802025-01-10T05:17:37.544ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:63450 req_id = 94c11070-733f-45a2-ac1b-ce895d701e8f resource = Database result = Ok(()) uri = /metrics/collectors/b6508fc7-5132-4a68-971d-9bf335b84514/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkNTAzY2RhOC01Mjc5LTRhNGMtOTFlNi02NjNjMTcyM2U2NzYifX0%3D
19812025-01-10T05:17:37.546ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 2816 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:63450 req_id = 94c11070-733f-45a2-ac1b-ce895d701e8f response_code = 200 uri = /metrics/collectors/b6508fc7-5132-4a68-971d-9bf335b84514/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkNTAzY2RhOC01Mjc5LTRhNGMtOTFlNi02NjNjMTcyM2U2NzYifX0%3D
19822025-01-10T05:17:37.547ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 228 local_addr = [fd00:1122:3344:101::c]:57161 method = GET remote_addr = [fd00:1122:3344:101::d]:46894 req_id = 483ad014-a88d-4832-be38-55dc932b2133 response_code = 200 uri = /3ca89dd1-7ada-4fe6-8600-a84d41dfa9d8
19832025-01-10T05:17:42.655ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 72 reason = Timeout
19842025-01-10T05:17:42.655ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
19852025-01-10T05:17:42.655ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 72 reason = Timeout
19862025-01-10T05:17:42.655ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
19872025-01-10T05:17:42.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
19882025-01-10T05:17:42.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
19892025-01-10T05:17:42.657ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19902025-01-10T05:17:42.660ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
19912025-01-10T05:17:42.660ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
19922025-01-10T05:17:42.660ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
19932025-01-10T05:17:42.662ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
19942025-01-10T05:17:42.662ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
19952025-01-10T05:17:42.664ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19962025-01-10T05:17:42.667ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 43ea24da-5730-4a85-9a8e-76459f9b69b1, lookup_type: ById(43ea24da-5730-4a85-9a8e-76459f9b69b1) } result = Ok(())
19972025-01-10T05:17:42.667ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
19982025-01-10T05:17:42.667ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
19992025-01-10T05:17:42.678ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 original_time_created = 2025-01-10 05:05:43.319091 UTC target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 time_created = 2025-01-10 05:05:43.319091 UTC
20002025-01-10T05:17:42.678ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 23.658978ms iteration = 72 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"43ea24da-5730-4a85-9a8e-76459f9b69b1","time_created":"2025-01-10T05:05:43.319091Z"}
20012025-01-10T05:17:42.683ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
20022025-01-10T05:17:42.683ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
20032025-01-10T05:17:42.710ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 55.422629ms iteration = 72 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
20042025-01-10T05:17:47.547ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 306 local_addr = [fd00:1122:3344:101::c]:57161 method = GET remote_addr = [fd00:1122:3344:101::d]:46894 req_id = 9a20c225-fe0d-4db4-a436-0fadf800c1b7 response_code = 200 uri = /3ca89dd1-7ada-4fe6-8600-a84d41dfa9d8
20052025-01-10T05:17:52.655ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 73 reason = Timeout
20062025-01-10T05:17:52.655ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 25 reason = Timeout
20072025-01-10T05:17:52.655ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 13 reason = Timeout
20082025-01-10T05:17:52.655ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
20092025-01-10T05:17:52.655ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
20102025-01-10T05:17:52.655ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
20112025-01-10T05:17:52.655ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 13 reason = Timeout
20122025-01-10T05:17:52.655ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 15 reason = Timeout
20132025-01-10T05:17:52.655ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 13 reason = Timeout
20142025-01-10T05:17:52.655ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 13 reason = Timeout
20152025-01-10T05:17:52.655ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
20162025-01-10T05:17:52.655ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
20172025-01-10T05:17:52.655ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
20182025-01-10T05:17:52.655ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 27 reason = Timeout
20192025-01-10T05:17:52.655ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
20202025-01-10T05:17:52.655ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 13 reason = Timeout
20212025-01-10T05:17:52.655ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 13 reason = Timeout
20222025-01-10T05:17:52.655ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
20232025-01-10T05:17:52.655ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 25 reason = Timeout
20242025-01-10T05:17:52.655ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 25 reason = Timeout
20252025-01-10T05:17:52.655ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
20262025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
20272025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 13 reason = Timeout
20282025-01-10T05:17:52.656ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-01-10 05:07:52.655960611 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
20292025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
20302025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 25 reason = Timeout
20312025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
20322025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 25 reason = Timeout
20332025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
20342025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 25 reason = Timeout
20352025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
20362025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
20372025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 13 reason = Timeout
20382025-01-10T05:17:52.656ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
20392025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 73 reason = Timeout
20402025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
20412025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
20422025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 25 reason = Timeout
20432025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
20442025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 25 reason = Timeout
20452025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
20462025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 25 reason = Timeout
20472025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
20482025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 25 reason = Timeout
20492025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 13 reason = Timeout
20502025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
20512025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 25 reason = Timeout
20522025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 6 reason = Timeout
20532025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
20542025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 25 reason = Timeout
20552025-01-10T05:17:52.656ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
20562025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
20572025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
20582025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 13 reason = Timeout
20592025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
20602025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
20612025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 25 reason = Timeout
20622025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
20632025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 14 reason = Timeout
20642025-01-10T05:17:52.656ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:83 reason = blueprint disabled target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1
20652025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 13.61\u{b5}s iteration = 14 status = {"enabled":false,"target_id":"43ea24da-5730-4a85-9a8e-76459f9b69b1"}
20662025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 15 reason = Timeout
20672025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
20682025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
20692025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
20702025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
20712025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
20722025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
20732025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
20742025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
20752025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
20762025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
20772025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
20782025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
20792025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
20802025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
20812025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
20822025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
20832025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
20842025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
20852025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
20862025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
20872025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
20882025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
20892025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
20902025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
20912025-01-10T05:17:52.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
20922025-01-10T05:17:52.657ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("30a043aa-79b8-40f0-889b-b1aa4686508a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("95b462f2-3c97-4463-9c08-5404518f33d2.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("95b462f2-3c97-4463-9c08-5404518f33d2.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 27083, tv_nsec: 164673758 } })
20932025-01-10T05:17:52.657ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("53514ca5-28a5-4b76-afd8-362873acf1d4.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("b3737e73-ceea-4b83-966e-05b55558955a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("e93887bd-ffe8-46b8-ad6f-92e0665448ab.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("e93887bd-ffe8-46b8-ad6f-92e0665448ab.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 27083, tv_nsec: 164663148 } })
20942025-01-10T05:17:52.657ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 25 reason = Timeout
20952025-01-10T05:17:52.657ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
20962025-01-10T05:17:52.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
20972025-01-10T05:17:52.657ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
20982025-01-10T05:17:52.657ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
20992025-01-10T05:17:52.657ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.306662ms iteration = 13 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
21002025-01-10T05:17:52.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
21012025-01-10T05:17:52.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
21022025-01-10T05:17:52.657ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
21032025-01-10T05:17:52.657ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
21042025-01-10T05:17:52.657ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.516009ms iteration = 13 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353"]}
21052025-01-10T05:17:52.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
21062025-01-10T05:17:52.657ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
21072025-01-10T05:17:52.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
21082025-01-10T05:17:52.658ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21092025-01-10T05:17:52.659ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21102025-01-10T05:17:52.660ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.330151ms iteration = 25 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
21112025-01-10T05:17:52.660ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 5.095186ms iteration = 13 status = {"errors":[],"found_port_ok":[]}
21122025-01-10T05:17:52.662ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
21132025-01-10T05:17:52.662ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21142025-01-10T05:17:52.662ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21152025-01-10T05:17:52.663ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
21162025-01-10T05:17:52.663ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
21172025-01-10T05:17:52.663ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
21182025-01-10T05:17:52.663ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
21192025-01-10T05:17:52.663ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
21202025-01-10T05:17:52.664ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21212025-01-10T05:17:52.665ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21222025-01-10T05:17:52.665ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 27083, tv_nsec: 173746683 } })
21232025-01-10T05:17:52.666ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 27083, tv_nsec: 173906738 } })
21242025-01-10T05:17:52.666ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1103
21252025-01-10T05:17:52.666ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1103
21262025-01-10T05:17:52.666ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-01-10 05:07:52.655960611 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
21272025-01-10T05:17:52.666ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 10.599114ms iteration = 13 status = {"expiration":"2025-01-10T05:07:52.655960611Z","pruned":[]}
21282025-01-10T05:17:52.667ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
21292025-01-10T05:17:52.667ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
21302025-01-10T05:17:52.667ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
21312025-01-10T05:17:52.668ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
21322025-01-10T05:17:52.668ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
21332025-01-10T05:17:52.668ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
21342025-01-10T05:17:52.669ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
21352025-01-10T05:17:52.669ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
21362025-01-10T05:17:52.670ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21372025-01-10T05:17:52.671ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
21382025-01-10T05:17:52.671ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
21392025-01-10T05:17:52.671ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21402025-01-10T05:17:52.671ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
21412025-01-10T05:17:52.671ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
21422025-01-10T05:17:52.671ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
21432025-01-10T05:17:52.671ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
21442025-01-10T05:17:52.673ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
21452025-01-10T05:17:52.673ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
21462025-01-10T05:17:52.673ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
21472025-01-10T05:17:52.673ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
21482025-01-10T05:17:52.673ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
21492025-01-10T05:17:52.673ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
21502025-01-10T05:17:52.673ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
21512025-01-10T05:17:52.673ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
21522025-01-10T05:17:52.674ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21532025-01-10T05:17:52.677ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21542025-01-10T05:17:52.678ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
21552025-01-10T05:17:52.678ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
21562025-01-10T05:17:52.678ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
21572025-01-10T05:17:52.680ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
21582025-01-10T05:17:52.680ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
21592025-01-10T05:17:52.680ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
21602025-01-10T05:17:52.684ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21612025-01-10T05:17:52.686ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
21622025-01-10T05:17:52.686ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
21632025-01-10T05:17:52.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
21642025-01-10T05:17:52.687ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
21652025-01-10T05:17:52.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
21662025-01-10T05:17:52.688ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21672025-01-10T05:17:52.689ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21682025-01-10T05:17:52.689ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21692025-01-10T05:17:52.689ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21702025-01-10T05:17:52.691ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
21712025-01-10T05:17:52.691ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
21722025-01-10T05:17:52.691ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
21732025-01-10T05:17:52.691ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21742025-01-10T05:17:52.693ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
21752025-01-10T05:17:52.693ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
21762025-01-10T05:17:52.693ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
21772025-01-10T05:17:52.693ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
21782025-01-10T05:17:52.695ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
21792025-01-10T05:17:52.696ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
21802025-01-10T05:17:52.696ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21812025-01-10T05:17:52.696ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
21822025-01-10T05:17:52.696ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
21832025-01-10T05:17:52.696ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
21842025-01-10T05:17:52.696ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
21852025-01-10T05:17:52.699ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
21862025-01-10T05:17:52.700ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
21872025-01-10T05:17:52.700ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:171 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
21882025-01-10T05:17:52.700ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 44.964639ms iteration = 13 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
21892025-01-10T05:17:52.701ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 27083, tv_nsec: 209826834 } })
21902025-01-10T05:17:52.702ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1103 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08
21912025-01-10T05:17:52.703ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 47.119089ms iteration = 25 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
21922025-01-10T05:17:52.703ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21932025-01-10T05:17:52.703ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
21942025-01-10T05:17:52.704ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
21952025-01-10T05:17:52.708ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 43ea24da-5730-4a85-9a8e-76459f9b69b1, lookup_type: ById(43ea24da-5730-4a85-9a8e-76459f9b69b1) } result = Ok(())
21962025-01-10T05:17:52.708ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21972025-01-10T05:17:52.708ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21982025-01-10T05:17:52.710ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21992025-01-10T05:17:52.711ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22002025-01-10T05:17:52.713ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
22012025-01-10T05:17:52.713ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
22022025-01-10T05:17:52.713ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
22032025-01-10T05:17:52.713ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
22042025-01-10T05:17:52.713ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
22052025-01-10T05:17:52.713ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
22062025-01-10T05:17:52.716ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22072025-01-10T05:17:52.721ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
22082025-01-10T05:17:52.722ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
22092025-01-10T05:17:52.722ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
22102025-01-10T05:17:52.722ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22112025-01-10T05:17:52.722ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 65.671201ms iteration = 25 status = {"errors":[],"garbage_collect_requested":[]}
22122025-01-10T05:17:52.722ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
22132025-01-10T05:17:52.722ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
22142025-01-10T05:17:52.722ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
22152025-01-10T05:17:52.725ZDEBGnexus (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 68.881066ms iteration = 6 status = {"nsuccess":5}
22162025-01-10T05:17:52.730ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 74.708974ms iteration = 25 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
22172025-01-10T05:17:52.730ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 73.775864ms iteration = 25 status = {"errors":[],"finish_invoked_ok":[]}
22182025-01-10T05:17:52.732ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
22192025-01-10T05:17:52.732ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
22202025-01-10T05:17:52.732ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
22212025-01-10T05:17:52.732ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
22222025-01-10T05:17:52.732ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
22232025-01-10T05:17:52.733ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22242025-01-10T05:17:52.735ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
22252025-01-10T05:17:52.735ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
22262025-01-10T05:17:52.735ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
22272025-01-10T05:17:52.735ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
22282025-01-10T05:17:52.735ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22292025-01-10T05:17:52.736ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 original_time_created = 2025-01-10 05:05:43.319091 UTC target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 time_created = 2025-01-10 05:05:43.319091 UTC
22302025-01-10T05:17:52.736ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 80.385849ms iteration = 73 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"43ea24da-5730-4a85-9a8e-76459f9b69b1","time_created":"2025-01-10T05:05:43.319091Z"}
22312025-01-10T05:17:52.737ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
22322025-01-10T05:17:52.737ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
22332025-01-10T05:17:52.737ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
22342025-01-10T05:17:52.738ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
22352025-01-10T05:17:52.738ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
22362025-01-10T05:17:52.738ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
22372025-01-10T05:17:52.739ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
22382025-01-10T05:17:52.739ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = bca3dac0-9831-415b-86b5-c6c6cc12ed56 dns_zone_name = oxide.test found_so_far = 0 version = 2
22392025-01-10T05:17:52.739ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
22402025-01-10T05:17:52.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
22412025-01-10T05:17:52.740ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
22422025-01-10T05:17:52.741ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
22432025-01-10T05:17:52.741ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22442025-01-10T05:17:52.741ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
22452025-01-10T05:17:52.741ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 85.423022ms iteration = 25 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
22462025-01-10T05:17:52.742ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 86.539599ms iteration = 73 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
22472025-01-10T05:17:52.743ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
22482025-01-10T05:17:52.743ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
22492025-01-10T05:17:52.743ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
22502025-01-10T05:17:52.743ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
22512025-01-10T05:17:52.743ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
22522025-01-10T05:17:52.743ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
22532025-01-10T05:17:52.743ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
22542025-01-10T05:17:52.743ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
22552025-01-10T05:17:52.743ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
22562025-01-10T05:17:52.743ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 87.758128ms iteration = 13 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
22572025-01-10T05:17:52.744ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22582025-01-10T05:17:52.745ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
22592025-01-10T05:17:52.745ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
22602025-01-10T05:17:52.745ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
22612025-01-10T05:17:52.746ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
22622025-01-10T05:17:52.746ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
22632025-01-10T05:17:52.747ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22642025-01-10T05:17:52.747ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = bca3dac0-9831-415b-86b5-c6c6cc12ed56 dns_zone_name = oxide.test found_so_far = 1 version = 2
22652025-01-10T05:17:52.747ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
22662025-01-10T05:17:52.747ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-01-10 05:06:09.259013 UTC dns_group = external generation = 2
22672025-01-10T05:17:52.747ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 91.995575ms iteration = 13 status = {"generation":2}
22682025-01-10T05:17:52.749ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59, lookup_type: ById(a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59) } result = Ok(())
22692025-01-10T05:17:52.749ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
22702025-01-10T05:17:52.749ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
22712025-01-10T05:17:52.751ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
22722025-01-10T05:17:52.751ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
22732025-01-10T05:17:52.752ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
22742025-01-10T05:17:52.752ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
22752025-01-10T05:17:52.752ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
22762025-01-10T05:17:52.753ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 97.875206ms iteration = 25 status = 0
22772025-01-10T05:17:52.761ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
22782025-01-10T05:17:52.761ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
22792025-01-10T05:17:52.767ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
22802025-01-10T05:17:52.768ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
22812025-01-10T05:17:52.771ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22822025-01-10T05:17:52.773ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
22832025-01-10T05:17:52.773ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
22842025-01-10T05:17:52.773ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
22852025-01-10T05:17:52.777ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
22862025-01-10T05:17:52.777ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
22872025-01-10T05:17:52.777ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 121.625678ms iteration = 25 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
22882025-01-10T05:17:52.797ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 141.273195ms iteration = 25 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
22892025-01-10T05:17:52.797ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1111 zone_address = fd00:1122:3344:101::2
22902025-01-10T05:17:52.797ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
22912025-01-10T05:17:52.797ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
22922025-01-10T05:17:52.797ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "da558c7f-ae4f-4f9a-9cab-ec2d372919ce", "content-length": "26", "date": "Fri, 10 Jan 2025 05:17:52 GMT"} })
22932025-01-10T05:17:52.797ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1116 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
22942025-01-10T05:17:52.798ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
22952025-01-10T05:17:52.798ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1111 zone_address = fd00:1122:3344:101::2
22962025-01-10T05:17:52.798ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
22972025-01-10T05:17:52.798ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "02d6ac41-b03e-4cca-b8cf-9efb0ccb20ab", "content-length": "26", "date": "Fri, 10 Jan 2025 05:17:52 GMT"} })
22982025-01-10T05:17:52.798ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1116 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
22992025-01-10T05:17:52.800ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1148 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
23002025-01-10T05:17:52.800ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1148 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
23012025-01-10T05:17:52.801ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
23022025-01-10T05:17:52.801ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 70f09b61-dde8-4da2-af2f-06138d5a975e dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
23032025-01-10T05:17:52.801ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
23042025-01-10T05:17:52.802ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
23052025-01-10T05:17:52.804ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23062025-01-10T05:17:52.806ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
23072025-01-10T05:17:52.806ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
23082025-01-10T05:17:52.806ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
23092025-01-10T05:17:52.809ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 70f09b61-dde8-4da2-af2f-06138d5a975e dns_zone_name = control-plane.oxide.internal found_so_far = 51 version = 1
23102025-01-10T05:17:52.809ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
23112025-01-10T05:17:52.809ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-01-10 05:06:08.425967 UTC dns_group = internal generation = 1
23122025-01-10T05:17:52.809ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 153.657607ms iteration = 13 status = {"generation":1}
23132025-01-10T05:17:52.820ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1111 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 zone_address = fd00:1122:3344:101::2
23142025-01-10T05:17:52.820ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
23152025-01-10T05:17:52.821ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "a710bc68-4446-43bf-b4a3-ae0388f99205", "content-length": "26", "date": "Fri, 10 Jan 2025 05:17:52 GMT"} })
23162025-01-10T05:17:52.821ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1116 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
23172025-01-10T05:17:52.821ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-01-10T05:05:45.360170Z, time_modified: 2025-01-10T05:05:45.360170Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 48a76ac8-2f94-4847-be0f-1cb4bfb661aa, db_silo: Silo { identity: SiloIdentity { id: 48a76ac8-2f94-4847-be0f-1cb4bfb661aa, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-01-10T05:06:09.192249Z, time_modified: 2025-01-10T05:06:09.192249Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "0de51367feb2d6f1959542dbb73fb728546b6d8dd71a0ec2fd5083120a87d20f" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 48a76ac8-2f94-4847-be0f-1cb4bfb661aa, db_silo: Silo { identity: SiloIdentity { id: 48a76ac8-2f94-4847-be0f-1cb4bfb661aa, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-01-10T05:06:09.192249Z, time_modified: 2025-01-10T05:06:09.192249Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "0de51367feb2d6f1959542dbb73fb728546b6d8dd71a0ec2fd5083120a87d20f" }] }) }
23182025-01-10T05:17:52.821ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 165.686547ms iteration = 13 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"48a76ac8-2f94-4847-be0f-1cb4bfb661aa","tls_certs":["0de51367feb2d6f1959542dbb73fb728546b6d8dd71a0ec2fd5083120a87d20f"]}},"default_endpoint":{"silo_id":"48a76ac8-2f94-4847-be0f-1cb4bfb661aa","tls_certs":["0de51367feb2d6f1959542dbb73fb728546b6d8dd71a0ec2fd5083120a87d20f"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
23192025-01-10T05:17:52.822ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1148 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 79fde928-482f-41ab-aee3-3f2478145f08
23202025-01-10T05:17:52.833ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 177.312834ms iteration = 27 status = {"physical_disks_added":0}
23212025-01-10T05:17:52.834ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
23222025-01-10T05:17:52.834ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 178.137831ms iteration = 13 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
23232025-01-10T05:17:52.929ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
23242025-01-10T05:17:52.930ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
23252025-01-10T05:17:52.931ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23262025-01-10T05:17:52.933ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
23272025-01-10T05:17:52.935ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
23282025-01-10T05:17:52.935ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
23292025-01-10T05:17:52.935ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
23302025-01-10T05:17:52.936ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23312025-01-10T05:17:52.939ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
23322025-01-10T05:17:52.939ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
23332025-01-10T05:17:52.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
23342025-01-10T05:17:52.948ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
23352025-01-10T05:17:52.949ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "5717c413-739b-4d98-9f72-d4546f94edd5", "content-length": "2", "date": "Fri, 10 Jan 2025 05:17:52 GMT"} })
23362025-01-10T05:17:52.950ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
23372025-01-10T05:17:52.950ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
23382025-01-10T05:17:52.953ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 297.710352ms iteration = 25 status = null
23392025-01-10T05:17:52.963ZINFOnexus (BackgroundTasks): VPC route manager sled a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
23402025-01-10T05:17:52.963ZDEBGnexus (BackgroundTasks): client request SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
23412025-01-10T05:17:52.963ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 307.579231ms iteration = 25 status = {}
23422025-01-10T05:17:52.963ZDEBGnexus (BackgroundTasks): client response SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "068ef6ef-0bab-4ae5-8917-00ec294b7ce7", "content-length": "372", "date": "Fri, 10 Jan 2025 05:17:52 GMT"} })
23432025-01-10T05:17:52.963ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
23442025-01-10T05:17:52.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
23452025-01-10T05:17:52.995ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
23462025-01-10T05:17:52.996ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "76975544-0974-406f-9853-3d5ba7fa2aff", "date": "Fri, 10 Jan 2025 05:17:52 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
23472025-01-10T05:17:52.996ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
23482025-01-10T05:17:52.997ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "111281d4-3f73-447b-9e3a-7ac2f8618cf3", "date": "Fri, 10 Jan 2025 05:17:52 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
23492025-01-10T05:17:52.997ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
23502025-01-10T05:17:52.999ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
23512025-01-10T05:17:52.999ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 343.276961ms iteration = 15 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
23522025-01-10T05:17:53.012ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
23532025-01-10T05:17:53.012ZDEBGnexus (BackgroundTasks): client request SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
23542025-01-10T05:17:53.012ZDEBGnexus (BackgroundTasks): client response SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "114e4744-8ab2-41a7-8e28-701dcefb5e91", "date": "Fri, 10 Jan 2025 05:17:52 GMT"} })
23552025-01-10T05:17:53.012ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
23562025-01-10T05:17:53.013ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
23572025-01-10T05:17:53.015ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
23582025-01-10T05:17:53.015ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
23592025-01-10T05:17:53.015ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
23602025-01-10T05:17:53.015ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "bc5ebb47-d828-4cbd-9906-2ac3defebd08", "date": "Fri, 10 Jan 2025 05:17:53 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
23612025-01-10T05:17:53.015ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
23622025-01-10T05:17:53.017ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "9bd6a7db-6ca1-4fca-b0c7-3ff2402227d8", "date": "Fri, 10 Jan 2025 05:17:53 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
23632025-01-10T05:17:53.017ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
23642025-01-10T05:17:53.018ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "00bbaf24-e83a-4cae-9f4e-a3ce7c04c18c", "date": "Fri, 10 Jan 2025 05:17:53 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
23652025-01-10T05:17:53.018ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
23662025-01-10T05:17:53.020ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
23672025-01-10T05:17:53.020ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 364.363115ms iteration = 15 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
23682025-01-10T05:17:53.025ZDEBGnexus (BackgroundTasks): client request SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
23692025-01-10T05:17:53.026ZDEBGnexus (BackgroundTasks): client response SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc282b1e-50ac-4272-a687-15c1fca3dca6", "content-length": "691", "date": "Fri, 10 Jan 2025 05:17:52 GMT"} })
23702025-01-10T05:17:53.026ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
23712025-01-10T05:17:53.026ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
23722025-01-10T05:17:53.026ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
23732025-01-10T05:17:53.027ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 371.31667ms iteration = 25 status = {}
23742025-01-10T05:17:53.028ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
23752025-01-10T05:17:53.029ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
23762025-01-10T05:17:53.032ZINFOnexus (BackgroundTasks): VPC route manager sled a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
23772025-01-10T05:17:53.032ZDEBGnexus (BackgroundTasks): client request SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
23782025-01-10T05:17:53.032ZDEBGnexus (BackgroundTasks): client response SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "4d4953e6-a056-4cb3-8416-89401c3cd1df", "date": "Fri, 10 Jan 2025 05:17:52 GMT"} })
23792025-01-10T05:17:53.034ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 378.08363ms iteration = 25 status = {}
23802025-01-10T05:17:53.073ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
23812025-01-10T05:17:53.073ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
23822025-01-10T05:17:53.074ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 5263359f-a80d-45e0-9595-0b15b904e78f, rack_id: 79fde928-482f-41ab-aee3-3f2478145f08, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(77858eb2-1069-4750-be5e-f3dafe20063a) } rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 switch_location = Switch0
23832025-01-10T05:17:53.075ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
23842025-01-10T05:17:53.075ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
23852025-01-10T05:17:53.084ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
23862025-01-10T05:17:53.084ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "fff31013-cb7b-4536-ba10-a1bab892aaff", "content-length": "134", "date": "Fri, 10 Jan 2025 05:17:52 GMT"} })
23872025-01-10T05:17:53.084ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
23882025-01-10T05:17:53.084ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 77858eb2-1069-4750-be5e-f3dafe20063a
23892025-01-10T05:17:53.084ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
23902025-01-10T05:17:53.084ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
23912025-01-10T05:17:53.084ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
23922025-01-10T05:17:53.084ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
23932025-01-10T05:17:53.084ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
23942025-01-10T05:17:53.084ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
23952025-01-10T05:17:53.084ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
23962025-01-10T05:17:53.084ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
23972025-01-10T05:17:53.085ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08
23982025-01-10T05:17:53.085ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
23992025-01-10T05:17:53.085ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24002025-01-10T05:17:53.087ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
24012025-01-10T05:17:53.087ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d25f130-a9d2-4467-8070-15031d34262f", "content-length": "2", "date": "Fri, 10 Jan 2025 05:17:52 GMT"} })
24022025-01-10T05:17:53.087ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
24032025-01-10T05:17:53.087ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "7363844a-fd1b-4433-b13f-b3f69d0cc987", "content-length": "52", "date": "Fri, 10 Jan 2025 05:17:52 GMT"} })
24042025-01-10T05:17:53.087ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 switch_location = Switch0
24052025-01-10T05:17:53.087ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
24062025-01-10T05:17:53.198ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "4dadb9d7-3afc-4a1a-ba3f-e08db5c1eb33", "date": "Fri, 10 Jan 2025 05:17:52 GMT"} })
24072025-01-10T05:17:53.198ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 switch = Switch0
24082025-01-10T05:17:53.199ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
24092025-01-10T05:17:53.199ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ec325f0-1579-4c6f-bf63-1244c3d33d40", "content-length": "682", "date": "Fri, 10 Jan 2025 05:17:53 GMT"} })
24102025-01-10T05:17:53.199ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24112025-01-10T05:17:53.199ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24122025-01-10T05:17:53.204ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24132025-01-10T05:17:53.205ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24142025-01-10T05:17:53.211ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24152025-01-10T05:17:53.211ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24162025-01-10T05:17:53.213ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24172025-01-10T05:17:53.214ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24182025-01-10T05:17:53.221ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 565.008827ms iteration = 25 status = {}
24192025-01-10T05:17:57.547ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 280 local_addr = [fd00:1122:3344:101::c]:57161 method = GET remote_addr = [fd00:1122:3344:101::d]:46894 req_id = 403b0df6-5d62-40a2-8762-c5e760189f19 response_code = 200 uri = /3ca89dd1-7ada-4fe6-8600-a84d41dfa9d8
24202025-01-10T05:18:02.655ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 74 reason = Timeout
24212025-01-10T05:18:02.655ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
24222025-01-10T05:18:02.655ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 74 reason = Timeout
24232025-01-10T05:18:02.655ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24242025-01-10T05:18:02.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24252025-01-10T05:18:02.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
24262025-01-10T05:18:02.657ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24272025-01-10T05:18:02.657ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
24282025-01-10T05:18:02.658ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
24292025-01-10T05:18:02.659ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.479715ms iteration = 74 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
24302025-01-10T05:18:02.660ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
24312025-01-10T05:18:02.660ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24322025-01-10T05:18:02.660ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24332025-01-10T05:18:02.661ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24342025-01-10T05:18:02.661ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24352025-01-10T05:18:02.662ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24362025-01-10T05:18:02.665ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 43ea24da-5730-4a85-9a8e-76459f9b69b1, lookup_type: ById(43ea24da-5730-4a85-9a8e-76459f9b69b1) } result = Ok(())
24372025-01-10T05:18:02.665ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24382025-01-10T05:18:02.665ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24392025-01-10T05:18:02.673ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 original_time_created = 2025-01-10 05:05:43.319091 UTC target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 time_created = 2025-01-10 05:05:43.319091 UTC
24402025-01-10T05:18:02.673ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 18.228411ms iteration = 74 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"43ea24da-5730-4a85-9a8e-76459f9b69b1","time_created":"2025-01-10T05:05:43.319091Z"}
24412025-01-10T05:18:07.540ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:63450 req_id = 17f385b5-e4f2-4031-ac52-8dfae4028985 roles = RoleSet { roles: {} } uri = /metrics/collectors/b6508fc7-5132-4a68-971d-9bf335b84514/producers?sort_by=id_ascending
24422025-01-10T05:18:07.540ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:63450 req_id = 17f385b5-e4f2-4031-ac52-8dfae4028985 resource = Database result = Ok(()) uri = /metrics/collectors/b6508fc7-5132-4a68-971d-9bf335b84514/producers?sort_by=id_ascending
24432025-01-10T05:18:07.543ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 3126 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:63450 req_id = 17f385b5-e4f2-4031-ac52-8dfae4028985 response_code = 200 uri = /metrics/collectors/b6508fc7-5132-4a68-971d-9bf335b84514/producers?sort_by=id_ascending
24442025-01-10T05:18:07.543ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:63450 req_id = 8a322dd2-0ba7-4475-ab3d-aaae1c5cd19a roles = RoleSet { roles: {} } uri = /metrics/collectors/b6508fc7-5132-4a68-971d-9bf335b84514/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkNTAzY2RhOC01Mjc5LTRhNGMtOTFlNi02NjNjMTcyM2U2NzYifX0%3D
24452025-01-10T05:18:07.543ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:63450 req_id = 8a322dd2-0ba7-4475-ab3d-aaae1c5cd19a resource = Database result = Ok(()) uri = /metrics/collectors/b6508fc7-5132-4a68-971d-9bf335b84514/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkNTAzY2RhOC01Mjc5LTRhNGMtOTFlNi02NjNjMTcyM2U2NzYifX0%3D
24462025-01-10T05:18:07.545ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 2382 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:63450 req_id = 8a322dd2-0ba7-4475-ab3d-aaae1c5cd19a response_code = 200 uri = /metrics/collectors/b6508fc7-5132-4a68-971d-9bf335b84514/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkNTAzY2RhOC01Mjc5LTRhNGMtOTFlNi02NjNjMTcyM2U2NzYifX0%3D
24472025-01-10T05:18:07.547ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 259 local_addr = [fd00:1122:3344:101::c]:57161 method = GET remote_addr = [fd00:1122:3344:101::d]:46894 req_id = 910673e7-280a-4057-a925-8590c0410da0 response_code = 200 uri = /3ca89dd1-7ada-4fe6-8600-a84d41dfa9d8
24482025-01-10T05:18:12.654ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 75 reason = Timeout
24492025-01-10T05:18:12.655ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 75 reason = Timeout
24502025-01-10T05:18:12.655ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24512025-01-10T05:18:12.655ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
24522025-01-10T05:18:12.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24532025-01-10T05:18:12.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
24542025-01-10T05:18:12.657ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24552025-01-10T05:18:12.657ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
24562025-01-10T05:18:12.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
24572025-01-10T05:18:12.659ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.92139ms iteration = 75 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
24582025-01-10T05:18:12.660ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
24592025-01-10T05:18:12.660ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24602025-01-10T05:18:12.660ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24612025-01-10T05:18:12.661ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24622025-01-10T05:18:12.662ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24632025-01-10T05:18:12.663ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24642025-01-10T05:18:12.666ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 43ea24da-5730-4a85-9a8e-76459f9b69b1, lookup_type: ById(43ea24da-5730-4a85-9a8e-76459f9b69b1) } result = Ok(())
24652025-01-10T05:18:12.666ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24662025-01-10T05:18:12.666ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24672025-01-10T05:18:12.675ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 original_time_created = 2025-01-10 05:05:43.319091 UTC target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 time_created = 2025-01-10 05:05:43.319091 UTC
24682025-01-10T05:18:12.675ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.225457ms iteration = 75 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"43ea24da-5730-4a85-9a8e-76459f9b69b1","time_created":"2025-01-10T05:05:43.319091Z"}
24692025-01-10T05:18:17.547ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 298 local_addr = [fd00:1122:3344:101::c]:57161 method = GET remote_addr = [fd00:1122:3344:101::d]:46894 req_id = d8a2b8e6-0168-4709-b3eb-5bbac0bb024b response_code = 200 uri = /3ca89dd1-7ada-4fe6-8600-a84d41dfa9d8
24702025-01-10T05:18:22.539ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:63450 req_id = a631bc0a-5171-4a42-9555-f54d28aecc5f roles = RoleSet { roles: {} } uri = /metrics/collectors/b6508fc7-5132-4a68-971d-9bf335b84514/producers?sort_by=id_ascending
24712025-01-10T05:18:22.540ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:63450 req_id = a631bc0a-5171-4a42-9555-f54d28aecc5f resource = Database result = Ok(()) uri = /metrics/collectors/b6508fc7-5132-4a68-971d-9bf335b84514/producers?sort_by=id_ascending
24722025-01-10T05:18:22.542ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 2335 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:63450 req_id = a631bc0a-5171-4a42-9555-f54d28aecc5f response_code = 200 uri = /metrics/collectors/b6508fc7-5132-4a68-971d-9bf335b84514/producers?sort_by=id_ascending
24732025-01-10T05:18:22.542ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:63450 req_id = 1c848ffd-8165-4c01-852b-ac1469de11e1 roles = RoleSet { roles: {} } uri = /metrics/collectors/b6508fc7-5132-4a68-971d-9bf335b84514/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkNTAzY2RhOC01Mjc5LTRhNGMtOTFlNi02NjNjMTcyM2U2NzYifX0%3D
24742025-01-10T05:18:22.542ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:63450 req_id = 1c848ffd-8165-4c01-852b-ac1469de11e1 resource = Database result = Ok(()) uri = /metrics/collectors/b6508fc7-5132-4a68-971d-9bf335b84514/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkNTAzY2RhOC01Mjc5LTRhNGMtOTFlNi02NjNjMTcyM2U2NzYifX0%3D
24752025-01-10T05:18:22.544ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 2379 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:63450 req_id = 1c848ffd-8165-4c01-852b-ac1469de11e1 response_code = 200 uri = /metrics/collectors/b6508fc7-5132-4a68-971d-9bf335b84514/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkNTAzY2RhOC01Mjc5LTRhNGMtOTFlNi02NjNjMTcyM2U2NzYifX0%3D
24762025-01-10T05:18:22.654ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 76 reason = Timeout
24772025-01-10T05:18:22.654ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 76 reason = Timeout
24782025-01-10T05:18:22.654ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
24792025-01-10T05:18:22.654ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 26 reason = Timeout
24802025-01-10T05:18:22.654ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24812025-01-10T05:18:22.654ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
24822025-01-10T05:18:22.654ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 28 reason = Timeout
24832025-01-10T05:18:22.654ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 26 reason = Timeout
24842025-01-10T05:18:22.654ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
24852025-01-10T05:18:22.654ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
24862025-01-10T05:18:22.654ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 26 reason = Timeout
24872025-01-10T05:18:22.654ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
24882025-01-10T05:18:22.654ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 26 reason = Timeout
24892025-01-10T05:18:22.654ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
24902025-01-10T05:18:22.654ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
24912025-01-10T05:18:22.654ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 26 reason = Timeout
24922025-01-10T05:18:22.654ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
24932025-01-10T05:18:22.654ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 26 reason = Timeout
24942025-01-10T05:18:22.654ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 26 reason = Timeout
24952025-01-10T05:18:22.654ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
24962025-01-10T05:18:22.654ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 26 reason = Timeout
24972025-01-10T05:18:22.654ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
24982025-01-10T05:18:22.654ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
24992025-01-10T05:18:22.654ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 26 reason = Timeout
25002025-01-10T05:18:22.654ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 26 reason = Timeout
25012025-01-10T05:18:22.655ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
25022025-01-10T05:18:22.655ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
25032025-01-10T05:18:22.655ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 26 reason = Timeout
25042025-01-10T05:18:22.655ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 26 reason = Timeout
25052025-01-10T05:18:22.655ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
25062025-01-10T05:18:22.655ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
25072025-01-10T05:18:22.655ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 26 reason = Timeout
25082025-01-10T05:18:22.655ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
25092025-01-10T05:18:22.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25102025-01-10T05:18:22.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
25112025-01-10T05:18:22.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
25122025-01-10T05:18:22.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
25132025-01-10T05:18:22.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
25142025-01-10T05:18:22.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
25152025-01-10T05:18:22.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
25162025-01-10T05:18:22.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
25172025-01-10T05:18:22.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
25182025-01-10T05:18:22.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
25192025-01-10T05:18:22.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
25202025-01-10T05:18:22.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
25212025-01-10T05:18:22.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
25222025-01-10T05:18:22.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
25232025-01-10T05:18:22.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
25242025-01-10T05:18:22.657ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 26 reason = Timeout
25252025-01-10T05:18:22.657ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
25262025-01-10T05:18:22.657ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
25272025-01-10T05:18:22.657ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25282025-01-10T05:18:22.657ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
25292025-01-10T05:18:22.658ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
25302025-01-10T05:18:22.658ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
25312025-01-10T05:18:22.658ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
25322025-01-10T05:18:22.658ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.147905ms iteration = 26 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
25332025-01-10T05:18:22.658ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25342025-01-10T05:18:22.660ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
25352025-01-10T05:18:22.660ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25362025-01-10T05:18:22.661ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25372025-01-10T05:18:22.661ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
25382025-01-10T05:18:22.661ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
25392025-01-10T05:18:22.661ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
25402025-01-10T05:18:22.662ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
25412025-01-10T05:18:22.662ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
25422025-01-10T05:18:22.662ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25432025-01-10T05:18:22.662ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25442025-01-10T05:18:22.664ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 27113, tv_nsec: 172356988 } })
25452025-01-10T05:18:22.664ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1103
25462025-01-10T05:18:22.664ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
25472025-01-10T05:18:22.664ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
25482025-01-10T05:18:22.664ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
25492025-01-10T05:18:22.664ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
25502025-01-10T05:18:22.665ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
25512025-01-10T05:18:22.665ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
25522025-01-10T05:18:22.665ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
25532025-01-10T05:18:22.666ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
25542025-01-10T05:18:22.666ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
25552025-01-10T05:18:22.666ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
25562025-01-10T05:18:22.666ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
25572025-01-10T05:18:22.667ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
25582025-01-10T05:18:22.667ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 27113, tv_nsec: 175640164 } })
25592025-01-10T05:18:22.667ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1103
25602025-01-10T05:18:22.668ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25612025-01-10T05:18:22.669ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 12.085372ms iteration = 26 status = {"errors":[],"finish_invoked_ok":[]}
25622025-01-10T05:18:22.670ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
25632025-01-10T05:18:22.670ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
25642025-01-10T05:18:22.670ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
25652025-01-10T05:18:22.670ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25662025-01-10T05:18:22.671ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25672025-01-10T05:18:22.672ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 17.612302ms iteration = 26 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
25682025-01-10T05:18:22.672ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
25692025-01-10T05:18:22.672ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
25702025-01-10T05:18:22.674ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
25712025-01-10T05:18:22.674ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
25722025-01-10T05:18:22.674ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 19.981518ms iteration = 26 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
25732025-01-10T05:18:22.676ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25742025-01-10T05:18:22.678ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
25752025-01-10T05:18:22.678ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
25762025-01-10T05:18:22.678ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
25772025-01-10T05:18:22.694ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
25782025-01-10T05:18:22.694ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
25792025-01-10T05:18:22.697ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 42.608013ms iteration = 26 status = {"errors":[],"garbage_collect_requested":[]}
25802025-01-10T05:18:22.697ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
25812025-01-10T05:18:22.697ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
25822025-01-10T05:18:22.702ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
25832025-01-10T05:18:22.702ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
25842025-01-10T05:18:22.704ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25852025-01-10T05:18:22.706ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
25862025-01-10T05:18:22.706ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
25872025-01-10T05:18:22.706ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
25882025-01-10T05:18:22.706ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25892025-01-10T05:18:22.708ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 53.60989ms iteration = 76 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
25902025-01-10T05:18:22.708ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
25912025-01-10T05:18:22.708ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
25922025-01-10T05:18:22.708ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
25932025-01-10T05:18:22.708ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 27113, tv_nsec: 217245055 } })
25942025-01-10T05:18:22.709ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1103 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08
25952025-01-10T05:18:22.709ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
25962025-01-10T05:18:22.709ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 54.930622ms iteration = 26 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
25972025-01-10T05:18:22.712ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25982025-01-10T05:18:22.715ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 43ea24da-5730-4a85-9a8e-76459f9b69b1, lookup_type: ById(43ea24da-5730-4a85-9a8e-76459f9b69b1) } result = Ok(())
25992025-01-10T05:18:22.715ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
26002025-01-10T05:18:22.715ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
26012025-01-10T05:18:22.716ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 61.91305ms iteration = 28 status = {"physical_disks_added":0}
26022025-01-10T05:18:22.719ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
26032025-01-10T05:18:22.719ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
26042025-01-10T05:18:22.719ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
26052025-01-10T05:18:22.719ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
26062025-01-10T05:18:22.721ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
26072025-01-10T05:18:22.721ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
26082025-01-10T05:18:22.721ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 66.593561ms iteration = 26 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
26092025-01-10T05:18:22.726ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 original_time_created = 2025-01-10 05:05:43.319091 UTC target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 time_created = 2025-01-10 05:05:43.319091 UTC
26102025-01-10T05:18:22.726ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 72.270025ms iteration = 76 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"43ea24da-5730-4a85-9a8e-76459f9b69b1","time_created":"2025-01-10T05:05:43.319091Z"}
26112025-01-10T05:18:22.727ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1111 zone_address = fd00:1122:3344:101::2
26122025-01-10T05:18:22.727ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
26132025-01-10T05:18:22.727ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b314d5d-b52c-48ad-9e34-d923afc98340", "content-length": "26", "date": "Fri, 10 Jan 2025 05:18:22 GMT"} })
26142025-01-10T05:18:22.727ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1116 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
26152025-01-10T05:18:22.728ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26162025-01-10T05:18:22.729ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1148 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
26172025-01-10T05:18:22.730ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 75.904063ms iteration = 26 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
26182025-01-10T05:18:22.730ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59, lookup_type: ById(a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59) } result = Ok(())
26192025-01-10T05:18:22.730ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
26202025-01-10T05:18:22.731ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
26212025-01-10T05:18:22.735ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 80.710659ms iteration = 26 status = 0
26222025-01-10T05:18:22.739ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1111 zone_address = fd00:1122:3344:101::2
26232025-01-10T05:18:22.739ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
26242025-01-10T05:18:22.739ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "502b9ab0-30dc-456a-9622-2143f907ace5", "content-length": "26", "date": "Fri, 10 Jan 2025 05:18:22 GMT"} })
26252025-01-10T05:18:22.739ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1116 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
26262025-01-10T05:18:22.741ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1148 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
26272025-01-10T05:18:22.751ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
26282025-01-10T05:18:22.752ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
26292025-01-10T05:18:22.759ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
26302025-01-10T05:18:22.760ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
26312025-01-10T05:18:22.761ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26322025-01-10T05:18:22.764ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
26332025-01-10T05:18:22.764ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
26342025-01-10T05:18:22.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
26352025-01-10T05:18:22.765ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26362025-01-10T05:18:22.767ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
26372025-01-10T05:18:22.768ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
26382025-01-10T05:18:22.768ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
26392025-01-10T05:18:22.815ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1111 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 zone_address = fd00:1122:3344:101::2
26402025-01-10T05:18:22.815ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
26412025-01-10T05:18:22.815ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "733f51fd-d672-41ce-b1d5-e03eb09c2824", "content-length": "26", "date": "Fri, 10 Jan 2025 05:18:22 GMT"} })
26422025-01-10T05:18:22.815ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1116 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
26432025-01-10T05:18:22.818ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1148 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 79fde928-482f-41ab-aee3-3f2478145f08
26442025-01-10T05:18:22.840ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
26452025-01-10T05:18:22.841ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8a121f7-377b-4415-82c6-15157e4068d0", "content-length": "2", "date": "Fri, 10 Jan 2025 05:18:22 GMT"} })
26462025-01-10T05:18:22.843ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
26472025-01-10T05:18:22.844ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
26482025-01-10T05:18:22.846ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 191.621608ms iteration = 26 status = null
26492025-01-10T05:18:22.871ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 216.586649ms iteration = 26 status = {}
26502025-01-10T05:18:22.892ZINFOnexus (BackgroundTasks): VPC route manager sled a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
26512025-01-10T05:18:22.892ZDEBGnexus (BackgroundTasks): client request SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
26522025-01-10T05:18:22.892ZDEBGnexus (BackgroundTasks): client response SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "4bb963d6-0f12-4d2c-a2fb-c7fb7e54e5d5", "content-length": "372", "date": "Fri, 10 Jan 2025 05:18:22 GMT"} })
26532025-01-10T05:18:22.893ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
26542025-01-10T05:18:22.893ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
26552025-01-10T05:18:22.894ZDEBGnexus (BackgroundTasks): client request SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
26562025-01-10T05:18:22.895ZDEBGnexus (BackgroundTasks): client response SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a3c025b-207a-4475-8560-f97366d5e893", "content-length": "691", "date": "Fri, 10 Jan 2025 05:18:22 GMT"} })
26572025-01-10T05:18:22.895ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
26582025-01-10T05:18:22.895ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
26592025-01-10T05:18:22.895ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
26602025-01-10T05:18:22.896ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 241.941522ms iteration = 26 status = {}
26612025-01-10T05:18:22.915ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
26622025-01-10T05:18:22.915ZDEBGnexus (BackgroundTasks): client request SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
26632025-01-10T05:18:22.916ZDEBGnexus (BackgroundTasks): client response SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "b1cdbce0-d234-4611-b189-e19a23a10961", "date": "Fri, 10 Jan 2025 05:18:22 GMT"} })
26642025-01-10T05:18:22.916ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
26652025-01-10T05:18:22.916ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
26662025-01-10T05:18:22.918ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
26672025-01-10T05:18:22.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
26682025-01-10T05:18:22.921ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
26692025-01-10T05:18:22.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
26702025-01-10T05:18:22.925ZINFOnexus (BackgroundTasks): VPC route manager sled a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
26712025-01-10T05:18:22.925ZDEBGnexus (BackgroundTasks): client request SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
26722025-01-10T05:18:22.926ZDEBGnexus (BackgroundTasks): client response SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "94f84228-330d-4742-b22d-dfaa231c92a7", "date": "Fri, 10 Jan 2025 05:18:22 GMT"} })
26732025-01-10T05:18:22.928ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 273.148194ms iteration = 26 status = {}
26742025-01-10T05:18:22.972ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
26752025-01-10T05:18:22.972ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
26762025-01-10T05:18:22.974ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 5263359f-a80d-45e0-9595-0b15b904e78f, rack_id: 79fde928-482f-41ab-aee3-3f2478145f08, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(77858eb2-1069-4750-be5e-f3dafe20063a) } rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 switch_location = Switch0
26772025-01-10T05:18:22.974ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
26782025-01-10T05:18:22.974ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
26792025-01-10T05:18:22.984ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
26802025-01-10T05:18:22.984ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "3abbc23c-7f02-4fe6-93e2-aa075b991f6c", "content-length": "134", "date": "Fri, 10 Jan 2025 05:18:22 GMT"} })
26812025-01-10T05:18:22.984ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
26822025-01-10T05:18:22.984ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 77858eb2-1069-4750-be5e-f3dafe20063a
26832025-01-10T05:18:22.985ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
26842025-01-10T05:18:22.985ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
26852025-01-10T05:18:22.985ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
26862025-01-10T05:18:22.985ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
26872025-01-10T05:18:22.985ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
26882025-01-10T05:18:22.985ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
26892025-01-10T05:18:22.985ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
26902025-01-10T05:18:22.985ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
26912025-01-10T05:18:22.985ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08
26922025-01-10T05:18:22.985ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
26932025-01-10T05:18:22.985ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
26942025-01-10T05:18:22.987ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
26952025-01-10T05:18:22.987ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "51b9126a-074f-434b-bbb2-97eeed9f534e", "content-length": "2", "date": "Fri, 10 Jan 2025 05:18:22 GMT"} })
26962025-01-10T05:18:22.987ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
26972025-01-10T05:18:22.987ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "b57bf5e8-c47d-403b-b51a-a239cda7070d", "content-length": "52", "date": "Fri, 10 Jan 2025 05:18:22 GMT"} })
26982025-01-10T05:18:22.987ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 switch_location = Switch0
26992025-01-10T05:18:22.987ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
27002025-01-10T05:18:23.087ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "55736207-51a4-412b-afae-206fa0af2bfc", "date": "Fri, 10 Jan 2025 05:18:22 GMT"} })
27012025-01-10T05:18:23.087ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 switch = Switch0
27022025-01-10T05:18:23.087ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
27032025-01-10T05:18:23.087ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a4331ed-a3e9-4043-878f-f0ad5740383d", "content-length": "682", "date": "Fri, 10 Jan 2025 05:18:22 GMT"} })
27042025-01-10T05:18:23.087ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
27052025-01-10T05:18:23.087ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
27062025-01-10T05:18:23.090ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
27072025-01-10T05:18:23.091ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
27082025-01-10T05:18:23.094ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
27092025-01-10T05:18:23.094ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
27102025-01-10T05:18:23.096ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
27112025-01-10T05:18:23.097ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
27122025-01-10T05:18:23.104ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 449.919402ms iteration = 26 status = {}
27132025-01-10T05:18:27.547ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 269 local_addr = [fd00:1122:3344:101::c]:57161 method = GET remote_addr = [fd00:1122:3344:101::d]:46894 req_id = 9191225e-8053-420d-bc87-ba09b491d2ae response_code = 200 uri = /3ca89dd1-7ada-4fe6-8600-a84d41dfa9d8
27142025-01-10T05:18:32.655ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 77 reason = Timeout
27152025-01-10T05:18:32.655ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
27162025-01-10T05:18:32.655ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 77 reason = Timeout
27172025-01-10T05:18:32.655ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
27182025-01-10T05:18:32.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
27192025-01-10T05:18:32.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
27202025-01-10T05:18:32.657ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27212025-01-10T05:18:32.658ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
27222025-01-10T05:18:32.658ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
27232025-01-10T05:18:32.659ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.572809ms iteration = 77 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
27242025-01-10T05:18:32.660ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
27252025-01-10T05:18:32.660ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
27262025-01-10T05:18:32.660ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
27272025-01-10T05:18:32.662ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
27282025-01-10T05:18:32.662ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
27292025-01-10T05:18:32.664ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27302025-01-10T05:18:32.666ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 43ea24da-5730-4a85-9a8e-76459f9b69b1, lookup_type: ById(43ea24da-5730-4a85-9a8e-76459f9b69b1) } result = Ok(())
27312025-01-10T05:18:32.666ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
27322025-01-10T05:18:32.666ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
27332025-01-10T05:18:32.677ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 original_time_created = 2025-01-10 05:05:43.319091 UTC target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 time_created = 2025-01-10 05:05:43.319091 UTC
27342025-01-10T05:18:32.677ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.246082ms iteration = 77 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"43ea24da-5730-4a85-9a8e-76459f9b69b1","time_created":"2025-01-10T05:05:43.319091Z"}
27352025-01-10T05:18:37.547ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 264 local_addr = [fd00:1122:3344:101::c]:57161 method = GET remote_addr = [fd00:1122:3344:101::d]:46894 req_id = 6eabcfff-1e01-4187-852e-7aca7a879dee response_code = 200 uri = /3ca89dd1-7ada-4fe6-8600-a84d41dfa9d8
27362025-01-10T05:18:42.654ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 78 reason = Timeout
27372025-01-10T05:18:42.654ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 78 reason = Timeout
27382025-01-10T05:18:42.654ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
27392025-01-10T05:18:42.654ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
27402025-01-10T05:18:42.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
27412025-01-10T05:18:42.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
27422025-01-10T05:18:42.656ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27432025-01-10T05:18:42.656ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
27442025-01-10T05:18:42.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
27452025-01-10T05:18:42.658ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.510607ms iteration = 78 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
27462025-01-10T05:18:42.659ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
27472025-01-10T05:18:42.659ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
27482025-01-10T05:18:42.659ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
27492025-01-10T05:18:42.661ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
27502025-01-10T05:18:42.661ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
27512025-01-10T05:18:42.662ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27522025-01-10T05:18:42.665ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 43ea24da-5730-4a85-9a8e-76459f9b69b1, lookup_type: ById(43ea24da-5730-4a85-9a8e-76459f9b69b1) } result = Ok(())
27532025-01-10T05:18:42.665ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
27542025-01-10T05:18:42.665ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
27552025-01-10T05:18:42.674ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 original_time_created = 2025-01-10 05:05:43.319091 UTC target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 time_created = 2025-01-10 05:05:43.319091 UTC
27562025-01-10T05:18:42.674ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.524926ms iteration = 78 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"43ea24da-5730-4a85-9a8e-76459f9b69b1","time_created":"2025-01-10T05:05:43.319091Z"}
27572025-01-10T05:18:47.547ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 274 local_addr = [fd00:1122:3344:101::c]:57161 method = GET remote_addr = [fd00:1122:3344:101::d]:46894 req_id = 7cd8a1bb-7618-4fd4-a5df-278e98f5ac88 response_code = 200 uri = /3ca89dd1-7ada-4fe6-8600-a84d41dfa9d8
27582025-01-10T05:18:52.540ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:63450 req_id = 6511e1b3-e6ca-459c-9d7a-acf54607924b roles = RoleSet { roles: {} } uri = /metrics/collectors/b6508fc7-5132-4a68-971d-9bf335b84514/producers?sort_by=id_ascending
27592025-01-10T05:18:52.540ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:63450 req_id = 6511e1b3-e6ca-459c-9d7a-acf54607924b resource = Database result = Ok(()) uri = /metrics/collectors/b6508fc7-5132-4a68-971d-9bf335b84514/producers?sort_by=id_ascending
27602025-01-10T05:18:52.541ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 1982 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:63450 req_id = 6511e1b3-e6ca-459c-9d7a-acf54607924b response_code = 200 uri = /metrics/collectors/b6508fc7-5132-4a68-971d-9bf335b84514/producers?sort_by=id_ascending
27612025-01-10T05:18:52.542ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:63450 req_id = 2ad0d422-9343-4f7b-af67-7e7a6d28b887 roles = RoleSet { roles: {} } uri = /metrics/collectors/b6508fc7-5132-4a68-971d-9bf335b84514/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkNTAzY2RhOC01Mjc5LTRhNGMtOTFlNi02NjNjMTcyM2U2NzYifX0%3D
27622025-01-10T05:18:52.542ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:63450 req_id = 2ad0d422-9343-4f7b-af67-7e7a6d28b887 resource = Database result = Ok(()) uri = /metrics/collectors/b6508fc7-5132-4a68-971d-9bf335b84514/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkNTAzY2RhOC01Mjc5LTRhNGMtOTFlNi02NjNjMTcyM2U2NzYifX0%3D
27632025-01-10T05:18:52.544ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 2273 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:63450 req_id = 2ad0d422-9343-4f7b-af67-7e7a6d28b887 response_code = 200 uri = /metrics/collectors/b6508fc7-5132-4a68-971d-9bf335b84514/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkNTAzY2RhOC01Mjc5LTRhNGMtOTFlNi02NjNjMTcyM2U2NzYifX0%3D
27642025-01-10T05:18:52.654ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 79 reason = Timeout
27652025-01-10T05:18:52.654ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 29 reason = Timeout
27662025-01-10T05:18:52.654ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
27672025-01-10T05:18:52.654ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
27682025-01-10T05:18:52.654ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 14 reason = Timeout
27692025-01-10T05:18:52.654ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
27702025-01-10T05:18:52.654ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 27 reason = Timeout
27712025-01-10T05:18:52.654ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 14 reason = Timeout
27722025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
27732025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
27742025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 27 reason = Timeout
27752025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 27 reason = Timeout
27762025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
27772025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
27782025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 14 reason = Timeout
27792025-01-10T05:18:52.655ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-01-10 05:08:52.654979906 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
27802025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 14 reason = Timeout
27812025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
27822025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
27832025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 27 reason = Timeout
27842025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
27852025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 27 reason = Timeout
27862025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
27872025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 27 reason = Timeout
27882025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
27892025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 27 reason = Timeout
27902025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
27912025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
27922025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 14 reason = Timeout
27932025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 14 reason = Timeout
27942025-01-10T05:18:52.655ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
27952025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
27962025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 27 reason = Timeout
27972025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
27982025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 27 reason = Timeout
27992025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
28002025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 14 reason = Timeout
28012025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 27 reason = Timeout
28022025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 79 reason = Timeout
28032025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
28042025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
28052025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 14 reason = Timeout
28062025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
28072025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
28082025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 27 reason = Timeout
28092025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 16 reason = Timeout
28102025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 14 reason = Timeout
28112025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
28122025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
28132025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 14 reason = Timeout
28142025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 27 reason = Timeout
28152025-01-10T05:18:52.655ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
28162025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 16 reason = Timeout
28172025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
28182025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
28192025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 27 reason = Timeout
28202025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
28212025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 15 reason = Timeout
28222025-01-10T05:18:52.655ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:83 reason = blueprint disabled target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1
28232025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 12.261\u{b5}s iteration = 15 status = {"enabled":false,"target_id":"43ea24da-5730-4a85-9a8e-76459f9b69b1"}
28242025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
28252025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
28262025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
28272025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
28282025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
28292025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
28302025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
28312025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
28322025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
28332025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
28342025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
28352025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
28362025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
28372025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
28382025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
28392025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
28402025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
28412025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
28422025-01-10T05:18:52.655ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("30a043aa-79b8-40f0-889b-b1aa4686508a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("95b462f2-3c97-4463-9c08-5404518f33d2.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("95b462f2-3c97-4463-9c08-5404518f33d2.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 27143, tv_nsec: 164463062 } })
28432025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
28442025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
28452025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
28462025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
28472025-01-10T05:18:52.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
28482025-01-10T05:18:52.655ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("53514ca5-28a5-4b76-afd8-362873acf1d4.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("b3737e73-ceea-4b83-966e-05b55558955a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("e93887bd-ffe8-46b8-ad6f-92e0665448ab.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("e93887bd-ffe8-46b8-ad6f-92e0665448ab.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 27143, tv_nsec: 164501473 } })
28492025-01-10T05:18:52.656ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 27 reason = Timeout
28502025-01-10T05:18:52.656ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
28512025-01-10T05:18:52.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
28522025-01-10T05:18:52.656ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
28532025-01-10T05:18:52.656ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.383475ms iteration = 14 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
28542025-01-10T05:18:52.656ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
28552025-01-10T05:18:52.656ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.518199ms iteration = 14 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353"]}
28562025-01-10T05:18:52.658ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28572025-01-10T05:18:52.658ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.539585ms iteration = 27 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
28582025-01-10T05:18:52.658ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28592025-01-10T05:18:52.659ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28602025-01-10T05:18:52.659ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
28612025-01-10T05:18:52.659ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:171 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
28622025-01-10T05:18:52.659ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 4.786356ms iteration = 14 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
28632025-01-10T05:18:52.660ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
28642025-01-10T05:18:52.660ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
28652025-01-10T05:18:52.661ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
28662025-01-10T05:18:52.662ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
28672025-01-10T05:18:52.662ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
28682025-01-10T05:18:52.662ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
28692025-01-10T05:18:52.662ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
28702025-01-10T05:18:52.662ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
28712025-01-10T05:18:52.662ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
28722025-01-10T05:18:52.663ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-01-10 05:08:52.654979906 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
28732025-01-10T05:18:52.663ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 8.401883ms iteration = 14 status = {"expiration":"2025-01-10T05:08:52.654979906Z","pruned":[]}
28742025-01-10T05:18:52.665ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28752025-01-10T05:18:52.665ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 27143, tv_nsec: 174307091 } })
28762025-01-10T05:18:52.665ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1103
28772025-01-10T05:18:52.666ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28782025-01-10T05:18:52.667ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
28792025-01-10T05:18:52.667ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
28802025-01-10T05:18:52.667ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
28812025-01-10T05:18:52.667ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
28822025-01-10T05:18:52.668ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
28832025-01-10T05:18:52.668ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
28842025-01-10T05:18:52.669ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
28852025-01-10T05:18:52.669ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
28862025-01-10T05:18:52.669ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
28872025-01-10T05:18:52.670ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
28882025-01-10T05:18:52.670ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28892025-01-10T05:18:52.671ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 27143, tv_nsec: 179919173 } })
28902025-01-10T05:18:52.671ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
28912025-01-10T05:18:52.671ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1103
28922025-01-10T05:18:52.671ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
28932025-01-10T05:18:52.671ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28942025-01-10T05:18:52.672ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28952025-01-10T05:18:52.672ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 17.459247ms iteration = 14 status = {"errors":[],"found_port_ok":[]}
28962025-01-10T05:18:52.673ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
28972025-01-10T05:18:52.673ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
28982025-01-10T05:18:52.674ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
28992025-01-10T05:18:52.674ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
29002025-01-10T05:18:52.674ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
29012025-01-10T05:18:52.674ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
29022025-01-10T05:18:52.674ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
29032025-01-10T05:18:52.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
29042025-01-10T05:18:52.675ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
29052025-01-10T05:18:52.675ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
29062025-01-10T05:18:52.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
29072025-01-10T05:18:52.676ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
29082025-01-10T05:18:52.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
29092025-01-10T05:18:52.676ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
29102025-01-10T05:18:52.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
29112025-01-10T05:18:52.678ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 22.428258ms iteration = 27 status = {"errors":[],"finish_invoked_ok":[]}
29122025-01-10T05:18:52.679ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
29132025-01-10T05:18:52.679ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
29142025-01-10T05:18:52.680ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
29152025-01-10T05:18:52.680ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
29162025-01-10T05:18:52.680ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
29172025-01-10T05:18:52.680ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
29182025-01-10T05:18:52.681ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
29192025-01-10T05:18:52.681ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
29202025-01-10T05:18:52.683ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29212025-01-10T05:18:52.683ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 27143, tv_nsec: 192651916 } })
29222025-01-10T05:18:52.684ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1103 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08
29232025-01-10T05:18:52.685ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29242025-01-10T05:18:52.688ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
29252025-01-10T05:18:52.688ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 43ea24da-5730-4a85-9a8e-76459f9b69b1, lookup_type: ById(43ea24da-5730-4a85-9a8e-76459f9b69b1) } result = Ok(())
29262025-01-10T05:18:52.688ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
29272025-01-10T05:18:52.688ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
29282025-01-10T05:18:52.688ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
29292025-01-10T05:18:52.688ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
29302025-01-10T05:18:52.692ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
29312025-01-10T05:18:52.693ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
29322025-01-10T05:18:52.698ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 43.924825ms iteration = 27 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
29332025-01-10T05:18:52.709ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
29342025-01-10T05:18:52.709ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
29352025-01-10T05:18:52.709ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
29362025-01-10T05:18:52.709ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
29372025-01-10T05:18:52.712ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 57.181956ms iteration = 27 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
29382025-01-10T05:18:52.715ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 60.911277ms iteration = 27 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
29392025-01-10T05:18:52.717ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 original_time_created = 2025-01-10 05:05:43.319091 UTC target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 time_created = 2025-01-10 05:05:43.319091 UTC
29402025-01-10T05:18:52.717ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 62.642133ms iteration = 79 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"43ea24da-5730-4a85-9a8e-76459f9b69b1","time_created":"2025-01-10T05:05:43.319091Z"}
29412025-01-10T05:18:52.719ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29422025-01-10T05:18:52.721ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 66.827078ms iteration = 27 status = {"errors":[],"garbage_collect_requested":[]}
29432025-01-10T05:18:52.722ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
29442025-01-10T05:18:52.722ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
29452025-01-10T05:18:52.722ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
29462025-01-10T05:18:52.724ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
29472025-01-10T05:18:52.725ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
29482025-01-10T05:18:52.725ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
29492025-01-10T05:18:52.725ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
29502025-01-10T05:18:52.725ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
29512025-01-10T05:18:52.725ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
29522025-01-10T05:18:52.726ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
29532025-01-10T05:18:52.726ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
29542025-01-10T05:18:52.727ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 72.693659ms iteration = 79 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
29552025-01-10T05:18:52.734ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29562025-01-10T05:18:52.735ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
29572025-01-10T05:18:52.735ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
29582025-01-10T05:18:52.735ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
29592025-01-10T05:18:52.735ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
29602025-01-10T05:18:52.735ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
29612025-01-10T05:18:52.737ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
29622025-01-10T05:18:52.737ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
29632025-01-10T05:18:52.737ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
29642025-01-10T05:18:52.737ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
29652025-01-10T05:18:52.737ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 82.150767ms iteration = 14 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
29662025-01-10T05:18:52.737ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
29672025-01-10T05:18:52.737ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29682025-01-10T05:18:52.739ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
29692025-01-10T05:18:52.739ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 84.11858ms iteration = 27 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
29702025-01-10T05:18:52.739ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
29712025-01-10T05:18:52.739ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
29722025-01-10T05:18:52.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
29732025-01-10T05:18:52.741ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29742025-01-10T05:18:52.744ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
29752025-01-10T05:18:52.746ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
29762025-01-10T05:18:52.746ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
29772025-01-10T05:18:52.746ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
29782025-01-10T05:18:52.746ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
29792025-01-10T05:18:52.746ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
29802025-01-10T05:18:52.746ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
29812025-01-10T05:18:52.746ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
29822025-01-10T05:18:52.787ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
29832025-01-10T05:18:52.788ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
29842025-01-10T05:18:52.795ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
29852025-01-10T05:18:52.804ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
29862025-01-10T05:18:52.833ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1111 zone_address = fd00:1122:3344:101::2
29872025-01-10T05:18:52.833ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
29882025-01-10T05:18:52.833ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "45d2f213-e9b6-4672-978c-76fec72d9e1d", "content-length": "26", "date": "Fri, 10 Jan 2025 05:18:52 GMT"} })
29892025-01-10T05:18:52.833ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1116 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
29902025-01-10T05:18:52.836ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1148 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
29912025-01-10T05:18:52.906ZINFOnexus (BackgroundTasks): VPC route manager sled a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
29922025-01-10T05:18:52.906ZDEBGnexus (BackgroundTasks): client request SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
29932025-01-10T05:18:52.907ZDEBGnexus (BackgroundTasks): client response SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "855c9431-6cc9-4588-857d-1f50f0751130", "content-length": "372", "date": "Fri, 10 Jan 2025 05:18:52 GMT"} })
29942025-01-10T05:18:52.907ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
29952025-01-10T05:18:52.907ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
29962025-01-10T05:18:52.909ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29972025-01-10T05:18:52.909ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29982025-01-10T05:18:52.914ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59, lookup_type: ById(a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59) } result = Ok(())
29992025-01-10T05:18:52.914ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
30002025-01-10T05:18:52.914ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
30012025-01-10T05:18:52.914ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
30022025-01-10T05:18:52.914ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
30032025-01-10T05:18:52.914ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
30042025-01-10T05:18:52.918ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30052025-01-10T05:18:52.920ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 265.030841ms iteration = 27 status = 0
30062025-01-10T05:18:52.921ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
30072025-01-10T05:18:52.921ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
30082025-01-10T05:18:52.921ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
30092025-01-10T05:18:52.925ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1111 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 zone_address = fd00:1122:3344:101::2
30102025-01-10T05:18:52.928ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
30112025-01-10T05:18:52.929ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "1aa733ba-2213-4a58-a4c6-535690d34bdf", "content-length": "26", "date": "Fri, 10 Jan 2025 05:18:52 GMT"} })
30122025-01-10T05:18:52.929ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1116 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
30132025-01-10T05:18:52.930ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30142025-01-10T05:18:52.933ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
30152025-01-10T05:18:52.933ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
30162025-01-10T05:18:52.933ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
30172025-01-10T05:18:52.936ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1148 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 79fde928-482f-41ab-aee3-3f2478145f08
30182025-01-10T05:18:52.953ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30192025-01-10T05:18:52.957ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
30202025-01-10T05:18:52.957ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
30212025-01-10T05:18:52.957ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
30222025-01-10T05:18:52.958ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
30232025-01-10T05:18:52.958ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
30242025-01-10T05:18:52.958ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 70f09b61-dde8-4da2-af2f-06138d5a975e dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
30252025-01-10T05:18:52.958ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
30262025-01-10T05:18:52.958ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
30272025-01-10T05:18:52.958ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
30282025-01-10T05:18:52.959ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1111 zone_address = fd00:1122:3344:101::2
30292025-01-10T05:18:52.959ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
30302025-01-10T05:18:52.959ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "f1d1c838-16f3-4048-972d-a5149346eebf", "content-length": "26", "date": "Fri, 10 Jan 2025 05:18:52 GMT"} })
30312025-01-10T05:18:52.959ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1116 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
30322025-01-10T05:18:52.960ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30332025-01-10T05:18:52.961ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1148 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
30342025-01-10T05:18:52.964ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
30352025-01-10T05:18:52.964ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
30362025-01-10T05:18:52.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
30372025-01-10T05:18:52.965ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
30382025-01-10T05:18:52.965ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
30392025-01-10T05:18:52.965ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 310.165966ms iteration = 27 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
30402025-01-10T05:18:52.992ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
30412025-01-10T05:18:52.992ZDEBGnexus (BackgroundTasks): client request SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
30422025-01-10T05:18:52.992ZDEBGnexus (BackgroundTasks): client response SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "3af3d55e-bd87-4082-b2c2-ad02f2430e3e", "date": "Fri, 10 Jan 2025 05:18:52 GMT"} })
30432025-01-10T05:18:52.992ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
30442025-01-10T05:18:52.993ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
30452025-01-10T05:18:52.995ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
30462025-01-10T05:18:52.995ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
30472025-01-10T05:18:52.997ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
30482025-01-10T05:18:52.997ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = bca3dac0-9831-415b-86b5-c6c6cc12ed56 dns_zone_name = oxide.test found_so_far = 0 version = 2
30492025-01-10T05:18:52.997ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
30502025-01-10T05:18:52.998ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
30512025-01-10T05:18:52.998ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30522025-01-10T05:18:52.999ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
30532025-01-10T05:18:52.999ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
30542025-01-10T05:18:53.000ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30552025-01-10T05:18:53.003ZINFOnexus (BackgroundTasks): VPC route manager sled a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
30562025-01-10T05:18:53.003ZDEBGnexus (BackgroundTasks): client request SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
30572025-01-10T05:18:53.003ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
30582025-01-10T05:18:53.003ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
30592025-01-10T05:18:53.003ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
30602025-01-10T05:18:53.003ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
30612025-01-10T05:18:53.003ZDEBGnexus (BackgroundTasks): client response SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "e107dd94-8217-4172-8aaa-0ef3c16a1540", "date": "Fri, 10 Jan 2025 05:18:52 GMT"} })
30622025-01-10T05:18:53.003ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
30632025-01-10T05:18:53.003ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
30642025-01-10T05:18:53.006ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 351.419705ms iteration = 27 status = {}
30652025-01-10T05:18:53.007ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 70f09b61-dde8-4da2-af2f-06138d5a975e dns_zone_name = control-plane.oxide.internal found_so_far = 51 version = 1
30662025-01-10T05:18:53.007ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
30672025-01-10T05:18:53.007ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-01-10 05:06:08.425967 UTC dns_group = internal generation = 1
30682025-01-10T05:18:53.007ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 352.53115ms iteration = 14 status = {"generation":1}
30692025-01-10T05:18:53.009ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = bca3dac0-9831-415b-86b5-c6c6cc12ed56 dns_zone_name = oxide.test found_so_far = 1 version = 2
30702025-01-10T05:18:53.009ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
30712025-01-10T05:18:53.009ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-01-10 05:06:09.259013 UTC dns_group = external generation = 2
30722025-01-10T05:18:53.009ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 354.074171ms iteration = 14 status = {"generation":2}
30732025-01-10T05:18:53.017ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-01-10T05:05:45.360170Z, time_modified: 2025-01-10T05:05:45.360170Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 48a76ac8-2f94-4847-be0f-1cb4bfb661aa, db_silo: Silo { identity: SiloIdentity { id: 48a76ac8-2f94-4847-be0f-1cb4bfb661aa, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-01-10T05:06:09.192249Z, time_modified: 2025-01-10T05:06:09.192249Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "0de51367feb2d6f1959542dbb73fb728546b6d8dd71a0ec2fd5083120a87d20f" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 48a76ac8-2f94-4847-be0f-1cb4bfb661aa, db_silo: Silo { identity: SiloIdentity { id: 48a76ac8-2f94-4847-be0f-1cb4bfb661aa, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-01-10T05:06:09.192249Z, time_modified: 2025-01-10T05:06:09.192249Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "0de51367feb2d6f1959542dbb73fb728546b6d8dd71a0ec2fd5083120a87d20f" }] }) }
30742025-01-10T05:18:53.017ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 362.774294ms iteration = 14 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"48a76ac8-2f94-4847-be0f-1cb4bfb661aa","tls_certs":["0de51367feb2d6f1959542dbb73fb728546b6d8dd71a0ec2fd5083120a87d20f"]}},"default_endpoint":{"silo_id":"48a76ac8-2f94-4847-be0f-1cb4bfb661aa","tls_certs":["0de51367feb2d6f1959542dbb73fb728546b6d8dd71a0ec2fd5083120a87d20f"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
30752025-01-10T05:18:53.027ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
30762025-01-10T05:18:53.027ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 372.043264ms iteration = 14 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
30772025-01-10T05:18:53.050ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 395.370391ms iteration = 29 status = {"physical_disks_added":0}
30782025-01-10T05:18:53.070ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
30792025-01-10T05:18:53.070ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "ab130f76-76dd-4f3d-ac6d-53be8885d0e7", "date": "Fri, 10 Jan 2025 05:18:52 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
30802025-01-10T05:18:53.070ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
30812025-01-10T05:18:53.073ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "f1dce964-e8d1-4a97-b389-628b0fb8b042", "date": "Fri, 10 Jan 2025 05:18:52 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
30822025-01-10T05:18:53.073ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
30832025-01-10T05:18:53.075ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
30842025-01-10T05:18:53.075ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 420.114424ms iteration = 16 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
30852025-01-10T05:18:53.085ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
30862025-01-10T05:18:53.161ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
30872025-01-10T05:18:53.161ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "58af0661-b77f-41d7-86aa-a61fd91d2789", "content-length": "2", "date": "Fri, 10 Jan 2025 05:18:52 GMT"} })
30882025-01-10T05:18:53.163ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
30892025-01-10T05:18:53.163ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
30902025-01-10T05:18:53.202ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 547.212919ms iteration = 27 status = {}
30912025-01-10T05:18:53.236ZDEBGnexus (BackgroundTasks): client request SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
30922025-01-10T05:18:53.236ZDEBGnexus (BackgroundTasks): client response SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "6054633f-6845-475e-a503-d33c1a3b5a94", "content-length": "691", "date": "Fri, 10 Jan 2025 05:18:52 GMT"} })
30932025-01-10T05:18:53.236ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
30942025-01-10T05:18:53.236ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
30952025-01-10T05:18:53.236ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
30962025-01-10T05:18:53.237ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
30972025-01-10T05:18:53.237ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 582.579907ms iteration = 27 status = null
30982025-01-10T05:18:53.237ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "773e2c72-3037-4420-9600-ecf1b3166c78", "date": "Fri, 10 Jan 2025 05:18:52 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
30992025-01-10T05:18:53.237ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
31002025-01-10T05:18:53.238ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 584.027893ms iteration = 27 status = {}
31012025-01-10T05:18:53.239ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "d470d803-186c-4db3-8e93-4a1d9a9a1cf8", "date": "Fri, 10 Jan 2025 05:18:53 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
31022025-01-10T05:18:53.239ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
31032025-01-10T05:18:53.241ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "ebe66b64-4c50-432e-9da8-7c513009b373", "date": "Fri, 10 Jan 2025 05:18:53 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
31042025-01-10T05:18:53.241ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
31052025-01-10T05:18:53.242ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
31062025-01-10T05:18:53.242ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 587.478596ms iteration = 16 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
31072025-01-10T05:18:53.267ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
31082025-01-10T05:18:53.267ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
31092025-01-10T05:18:53.269ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 5263359f-a80d-45e0-9595-0b15b904e78f, rack_id: 79fde928-482f-41ab-aee3-3f2478145f08, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(77858eb2-1069-4750-be5e-f3dafe20063a) } rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 switch_location = Switch0
31102025-01-10T05:18:53.269ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
31112025-01-10T05:18:53.269ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
31122025-01-10T05:18:53.278ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
31132025-01-10T05:18:53.278ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "c3873706-d4c2-46ac-92e7-6a0aab28dda2", "content-length": "134", "date": "Fri, 10 Jan 2025 05:18:52 GMT"} })
31142025-01-10T05:18:53.278ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
31152025-01-10T05:18:53.278ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 77858eb2-1069-4750-be5e-f3dafe20063a
31162025-01-10T05:18:53.278ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
31172025-01-10T05:18:53.278ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
31182025-01-10T05:18:53.278ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
31192025-01-10T05:18:53.278ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
31202025-01-10T05:18:53.278ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
31212025-01-10T05:18:53.278ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
31222025-01-10T05:18:53.279ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
31232025-01-10T05:18:53.279ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
31242025-01-10T05:18:53.279ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08
31252025-01-10T05:18:53.279ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
31262025-01-10T05:18:53.279ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
31272025-01-10T05:18:53.281ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
31282025-01-10T05:18:53.281ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "24400230-37f8-4b51-8018-f980fae543aa", "content-length": "2", "date": "Fri, 10 Jan 2025 05:18:52 GMT"} })
31292025-01-10T05:18:53.281ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
31302025-01-10T05:18:53.281ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "57ab3d4d-6066-4a90-bcfc-3c05e0ce58e1", "content-length": "52", "date": "Fri, 10 Jan 2025 05:18:52 GMT"} })
31312025-01-10T05:18:53.281ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 switch_location = Switch0
31322025-01-10T05:18:53.281ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
31332025-01-10T05:18:53.376ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "f47ec3d8-3c71-46e1-ac63-5279c17c7216", "date": "Fri, 10 Jan 2025 05:18:52 GMT"} })
31342025-01-10T05:18:53.376ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 switch = Switch0
31352025-01-10T05:18:53.376ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
31362025-01-10T05:18:53.380ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "d770df25-391b-4754-9439-f46ab61396df", "content-length": "682", "date": "Fri, 10 Jan 2025 05:18:52 GMT"} })
31372025-01-10T05:18:53.380ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
31382025-01-10T05:18:53.380ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
31392025-01-10T05:18:53.384ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
31402025-01-10T05:18:53.384ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
31412025-01-10T05:18:53.386ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
31422025-01-10T05:18:53.387ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
31432025-01-10T05:18:53.388ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
31442025-01-10T05:18:53.388ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
31452025-01-10T05:18:53.395ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 740.692638ms iteration = 27 status = {}
31462025-01-10T05:18:57.547ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 315 local_addr = [fd00:1122:3344:101::c]:57161 method = GET remote_addr = [fd00:1122:3344:101::d]:46894 req_id = 2f91c4c3-382c-45c5-904e-0bc8a2d11bd6 response_code = 200 uri = /3ca89dd1-7ada-4fe6-8600-a84d41dfa9d8
31472025-01-10T05:19:02.654ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 80 reason = Timeout
31482025-01-10T05:19:02.654ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 80 reason = Timeout
31492025-01-10T05:19:02.654ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
31502025-01-10T05:19:02.654ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
31512025-01-10T05:19:02.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
31522025-01-10T05:19:02.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
31532025-01-10T05:19:02.656ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31542025-01-10T05:19:02.659ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
31552025-01-10T05:19:02.659ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
31562025-01-10T05:19:02.659ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
31572025-01-10T05:19:02.661ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
31582025-01-10T05:19:02.661ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
31592025-01-10T05:19:02.663ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31602025-01-10T05:19:02.665ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 43ea24da-5730-4a85-9a8e-76459f9b69b1, lookup_type: ById(43ea24da-5730-4a85-9a8e-76459f9b69b1) } result = Ok(())
31612025-01-10T05:19:02.665ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
31622025-01-10T05:19:02.665ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
31632025-01-10T05:19:02.677ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 original_time_created = 2025-01-10 05:05:43.319091 UTC target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 time_created = 2025-01-10 05:05:43.319091 UTC
31642025-01-10T05:19:02.677ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 23.605826ms iteration = 80 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"43ea24da-5730-4a85-9a8e-76459f9b69b1","time_created":"2025-01-10T05:05:43.319091Z"}
31652025-01-10T05:19:02.684ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
31662025-01-10T05:19:02.684ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
31672025-01-10T05:19:02.712ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 58.112996ms iteration = 80 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
31682025-01-10T05:19:07.546ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 276 local_addr = [fd00:1122:3344:101::c]:57161 method = GET remote_addr = [fd00:1122:3344:101::d]:46894 req_id = f5271960-4eb5-4c42-8e46-db2950725790 response_code = 200 uri = /3ca89dd1-7ada-4fe6-8600-a84d41dfa9d8
31692025-01-10T05:19:12.653ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 81 reason = Timeout
31702025-01-10T05:19:12.653ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
31712025-01-10T05:19:12.653ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 81 reason = Timeout
31722025-01-10T05:19:12.653ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
31732025-01-10T05:19:12.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
31742025-01-10T05:19:12.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
31752025-01-10T05:19:12.656ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
31762025-01-10T05:19:12.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
31772025-01-10T05:19:12.656ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31782025-01-10T05:19:12.659ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
31792025-01-10T05:19:12.659ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
31802025-01-10T05:19:12.659ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
31812025-01-10T05:19:12.661ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
31822025-01-10T05:19:12.661ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
31832025-01-10T05:19:12.663ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31842025-01-10T05:19:12.665ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 43ea24da-5730-4a85-9a8e-76459f9b69b1, lookup_type: ById(43ea24da-5730-4a85-9a8e-76459f9b69b1) } result = Ok(())
31852025-01-10T05:19:12.665ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
31862025-01-10T05:19:12.665ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
31872025-01-10T05:19:12.677ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 original_time_created = 2025-01-10 05:05:43.319091 UTC target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 time_created = 2025-01-10 05:05:43.319091 UTC
31882025-01-10T05:19:12.677ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 23.266035ms iteration = 81 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"43ea24da-5730-4a85-9a8e-76459f9b69b1","time_created":"2025-01-10T05:05:43.319091Z"}
31892025-01-10T05:19:12.684ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 30.908643ms iteration = 81 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
31902025-01-10T05:19:17.546ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 264 local_addr = [fd00:1122:3344:101::c]:57161 method = GET remote_addr = [fd00:1122:3344:101::d]:46894 req_id = 5da25816-0e53-4c32-9649-aa468a2b0554 response_code = 200 uri = /3ca89dd1-7ada-4fe6-8600-a84d41dfa9d8
31912025-01-10T05:19:22.654ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 82 reason = Timeout
31922025-01-10T05:19:22.654ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 82 reason = Timeout
31932025-01-10T05:19:22.654ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
31942025-01-10T05:19:22.654ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
31952025-01-10T05:19:22.654ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 28 reason = Timeout
31962025-01-10T05:19:22.654ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
31972025-01-10T05:19:22.654ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
31982025-01-10T05:19:22.654ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 28 reason = Timeout
31992025-01-10T05:19:22.654ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
32002025-01-10T05:19:22.654ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 28 reason = Timeout
32012025-01-10T05:19:22.654ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
32022025-01-10T05:19:22.654ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 28 reason = Timeout
32032025-01-10T05:19:22.654ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
32042025-01-10T05:19:22.654ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 28 reason = Timeout
32052025-01-10T05:19:22.654ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
32062025-01-10T05:19:22.654ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
32072025-01-10T05:19:22.654ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 28 reason = Timeout
32082025-01-10T05:19:22.654ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 28 reason = Timeout
32092025-01-10T05:19:22.654ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
32102025-01-10T05:19:22.654ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
32112025-01-10T05:19:22.654ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 28 reason = Timeout
32122025-01-10T05:19:22.654ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
32132025-01-10T05:19:22.654ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 30 reason = Timeout
32142025-01-10T05:19:22.654ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
32152025-01-10T05:19:22.654ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 28 reason = Timeout
32162025-01-10T05:19:22.654ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
32172025-01-10T05:19:22.654ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 28 reason = Timeout
32182025-01-10T05:19:22.654ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
32192025-01-10T05:19:22.654ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 28 reason = Timeout
32202025-01-10T05:19:22.654ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
32212025-01-10T05:19:22.654ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 28 reason = Timeout
32222025-01-10T05:19:22.654ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
32232025-01-10T05:19:22.654ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 28 reason = Timeout
32242025-01-10T05:19:22.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
32252025-01-10T05:19:22.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
32262025-01-10T05:19:22.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
32272025-01-10T05:19:22.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
32282025-01-10T05:19:22.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
32292025-01-10T05:19:22.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
32302025-01-10T05:19:22.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
32312025-01-10T05:19:22.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
32322025-01-10T05:19:22.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
32332025-01-10T05:19:22.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
32342025-01-10T05:19:22.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
32352025-01-10T05:19:22.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
32362025-01-10T05:19:22.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
32372025-01-10T05:19:22.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
32382025-01-10T05:19:22.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
32392025-01-10T05:19:22.656ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 28 reason = Timeout
32402025-01-10T05:19:22.656ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
32412025-01-10T05:19:22.656ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32422025-01-10T05:19:22.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
32432025-01-10T05:19:22.657ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32442025-01-10T05:19:22.658ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.618278ms iteration = 28 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
32452025-01-10T05:19:22.658ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
32462025-01-10T05:19:22.658ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
32472025-01-10T05:19:22.658ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
32482025-01-10T05:19:22.659ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
32492025-01-10T05:19:22.660ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
32502025-01-10T05:19:22.660ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32512025-01-10T05:19:22.660ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
32522025-01-10T05:19:22.660ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
32532025-01-10T05:19:22.660ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
32542025-01-10T05:19:22.660ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
32552025-01-10T05:19:22.663ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32562025-01-10T05:19:22.663ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 27173, tv_nsec: 172893735 } })
32572025-01-10T05:19:22.664ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1103
32582025-01-10T05:19:22.664ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32592025-01-10T05:19:22.667ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
32602025-01-10T05:19:22.667ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
32612025-01-10T05:19:22.667ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
32622025-01-10T05:19:22.667ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
32632025-01-10T05:19:22.667ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
32642025-01-10T05:19:22.668ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
32652025-01-10T05:19:22.675ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 27173, tv_nsec: 184527293 } })
32662025-01-10T05:19:22.675ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1103
32672025-01-10T05:19:22.676ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
32682025-01-10T05:19:22.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
32692025-01-10T05:19:22.677ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
32702025-01-10T05:19:22.677ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
32712025-01-10T05:19:22.677ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32722025-01-10T05:19:22.678ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
32732025-01-10T05:19:22.691ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 36.950729ms iteration = 28 status = {"errors":[],"garbage_collect_requested":[]}
32742025-01-10T05:19:22.698ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
32752025-01-10T05:19:22.698ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
32762025-01-10T05:19:22.698ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
32772025-01-10T05:19:22.698ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
32782025-01-10T05:19:22.699ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
32792025-01-10T05:19:22.699ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
32802025-01-10T05:19:22.700ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
32812025-01-10T05:19:22.700ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
32822025-01-10T05:19:22.702ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 47.643006ms iteration = 28 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
32832025-01-10T05:19:22.702ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32842025-01-10T05:19:22.702ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
32852025-01-10T05:19:22.702ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
32862025-01-10T05:19:22.703ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32872025-01-10T05:19:22.704ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 48.150823ms iteration = 28 status = {"errors":[],"finish_invoked_ok":[]}
32882025-01-10T05:19:22.705ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32892025-01-10T05:19:22.705ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
32902025-01-10T05:19:22.705ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
32912025-01-10T05:19:22.705ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
32922025-01-10T05:19:22.707ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
32932025-01-10T05:19:22.707ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32942025-01-10T05:19:22.707ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59, lookup_type: ById(a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59) } result = Ok(())
32952025-01-10T05:19:22.707ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
32962025-01-10T05:19:22.707ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
32972025-01-10T05:19:22.707ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
32982025-01-10T05:19:22.707ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
32992025-01-10T05:19:22.707ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
33002025-01-10T05:19:22.707ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
33012025-01-10T05:19:22.707ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 27173, tv_nsec: 216907344 } })
33022025-01-10T05:19:22.707ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
33032025-01-10T05:19:22.707ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33042025-01-10T05:19:22.708ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1103 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08
33052025-01-10T05:19:22.708ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
33062025-01-10T05:19:22.708ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
33072025-01-10T05:19:22.709ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
33082025-01-10T05:19:22.710ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
33092025-01-10T05:19:22.710ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
33102025-01-10T05:19:22.710ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
33112025-01-10T05:19:22.710ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 55.905935ms iteration = 28 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
33122025-01-10T05:19:22.710ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
33132025-01-10T05:19:22.710ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 43ea24da-5730-4a85-9a8e-76459f9b69b1, lookup_type: ById(43ea24da-5730-4a85-9a8e-76459f9b69b1) } result = Ok(())
33142025-01-10T05:19:22.710ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
33152025-01-10T05:19:22.710ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
33162025-01-10T05:19:22.711ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
33172025-01-10T05:19:22.711ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
33182025-01-10T05:19:22.712ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 57.492515ms iteration = 28 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
33192025-01-10T05:19:22.712ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 58.018333ms iteration = 28 status = 0
33202025-01-10T05:19:22.713ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33212025-01-10T05:19:22.715ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
33222025-01-10T05:19:22.715ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
33232025-01-10T05:19:22.715ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
33242025-01-10T05:19:22.720ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 65.997612ms iteration = 30 status = {"physical_disks_added":0}
33252025-01-10T05:19:22.725ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 original_time_created = 2025-01-10 05:05:43.319091 UTC target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 time_created = 2025-01-10 05:05:43.319091 UTC
33262025-01-10T05:19:22.725ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 70.830899ms iteration = 82 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"43ea24da-5730-4a85-9a8e-76459f9b69b1","time_created":"2025-01-10T05:05:43.319091Z"}
33272025-01-10T05:19:22.730ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 75.650495ms iteration = 82 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
33282025-01-10T05:19:22.737ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1111 zone_address = fd00:1122:3344:101::2
33292025-01-10T05:19:22.737ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
33302025-01-10T05:19:22.737ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2b412d3-9a4f-449d-8e2b-d623541eb2a5", "content-length": "26", "date": "Fri, 10 Jan 2025 05:19:22 GMT"} })
33312025-01-10T05:19:22.737ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1116 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
33322025-01-10T05:19:22.739ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1148 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
33332025-01-10T05:19:22.742ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 87.658165ms iteration = 28 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
33342025-01-10T05:19:22.745ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
33352025-01-10T05:19:22.745ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
33362025-01-10T05:19:22.745ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 90.676682ms iteration = 28 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
33372025-01-10T05:19:22.766ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1111 zone_address = fd00:1122:3344:101::2
33382025-01-10T05:19:22.766ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
33392025-01-10T05:19:22.766ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e57ca97-dd8b-46c3-85c6-e74c8f8e74a8", "content-length": "26", "date": "Fri, 10 Jan 2025 05:19:22 GMT"} })
33402025-01-10T05:19:22.766ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1116 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
33412025-01-10T05:19:22.768ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1148 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
33422025-01-10T05:19:22.781ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
33432025-01-10T05:19:22.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
33442025-01-10T05:19:22.899ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
33452025-01-10T05:19:22.899ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
33462025-01-10T05:19:22.915ZINFOnexus (BackgroundTasks): VPC route manager sled a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
33472025-01-10T05:19:22.915ZDEBGnexus (BackgroundTasks): client request SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
33482025-01-10T05:19:22.915ZDEBGnexus (BackgroundTasks): client response SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "d2ef33ea-f7b1-465d-9441-0ea61c8e2360", "content-length": "372", "date": "Fri, 10 Jan 2025 05:19:22 GMT"} })
33492025-01-10T05:19:22.915ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
33502025-01-10T05:19:22.916ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
33512025-01-10T05:19:22.916ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33522025-01-10T05:19:22.918ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
33532025-01-10T05:19:22.918ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
33542025-01-10T05:19:22.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
33552025-01-10T05:19:22.927ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1111 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 zone_address = fd00:1122:3344:101::2
33562025-01-10T05:19:22.927ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
33572025-01-10T05:19:22.927ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "049a8ff6-2d49-47df-8557-fe9f97a00bc3", "content-length": "26", "date": "Fri, 10 Jan 2025 05:19:22 GMT"} })
33582025-01-10T05:19:22.927ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1116 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
33592025-01-10T05:19:22.930ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1148 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 79fde928-482f-41ab-aee3-3f2478145f08
33602025-01-10T05:19:22.931ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
33612025-01-10T05:19:22.931ZDEBGnexus (BackgroundTasks): client request SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
33622025-01-10T05:19:22.932ZDEBGnexus (BackgroundTasks): client response SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "1bbb377a-c426-46ce-8a7a-b2f7041f8ea7", "date": "Fri, 10 Jan 2025 05:19:22 GMT"} })
33632025-01-10T05:19:22.932ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
33642025-01-10T05:19:22.932ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
33652025-01-10T05:19:22.934ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
33662025-01-10T05:19:22.934ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
33672025-01-10T05:19:22.943ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
33682025-01-10T05:19:22.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
33692025-01-10T05:19:22.947ZINFOnexus (BackgroundTasks): VPC route manager sled a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
33702025-01-10T05:19:22.947ZDEBGnexus (BackgroundTasks): client request SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
33712025-01-10T05:19:22.947ZDEBGnexus (BackgroundTasks): client response SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "2bf44e43-9d45-4f1f-ab34-620c8714394c", "date": "Fri, 10 Jan 2025 05:19:22 GMT"} })
33722025-01-10T05:19:22.949ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 295.370656ms iteration = 28 status = {}
33732025-01-10T05:19:22.952ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33742025-01-10T05:19:22.954ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
33752025-01-10T05:19:22.954ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
33762025-01-10T05:19:22.954ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
33772025-01-10T05:19:23.006ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
33782025-01-10T05:19:23.007ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "23956260-3d61-449f-9e87-e02edc7e4700", "content-length": "2", "date": "Fri, 10 Jan 2025 05:19:22 GMT"} })
33792025-01-10T05:19:23.009ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
33802025-01-10T05:19:23.009ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
33812025-01-10T05:19:23.011ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 356.761529ms iteration = 28 status = null
33822025-01-10T05:19:23.017ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 362.425832ms iteration = 28 status = {}
33832025-01-10T05:19:23.038ZDEBGnexus (BackgroundTasks): client request SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
33842025-01-10T05:19:23.039ZDEBGnexus (BackgroundTasks): client response SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "0baa764a-77ff-4b5e-826b-5e35c518a909", "content-length": "691", "date": "Fri, 10 Jan 2025 05:19:22 GMT"} })
33852025-01-10T05:19:23.039ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
33862025-01-10T05:19:23.039ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
33872025-01-10T05:19:23.039ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
33882025-01-10T05:19:23.041ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 386.949748ms iteration = 28 status = {}
33892025-01-10T05:19:23.090ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
33902025-01-10T05:19:23.090ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
33912025-01-10T05:19:23.092ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 5263359f-a80d-45e0-9595-0b15b904e78f, rack_id: 79fde928-482f-41ab-aee3-3f2478145f08, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(77858eb2-1069-4750-be5e-f3dafe20063a) } rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 switch_location = Switch0
33922025-01-10T05:19:23.092ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
33932025-01-10T05:19:23.093ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
33942025-01-10T05:19:23.103ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
33952025-01-10T05:19:23.104ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "77751762-222d-435a-9dc2-27e89d65dbd0", "content-length": "134", "date": "Fri, 10 Jan 2025 05:19:23 GMT"} })
33962025-01-10T05:19:23.104ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
33972025-01-10T05:19:23.104ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 77858eb2-1069-4750-be5e-f3dafe20063a
33982025-01-10T05:19:23.104ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
33992025-01-10T05:19:23.104ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
34002025-01-10T05:19:23.104ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
34012025-01-10T05:19:23.104ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
34022025-01-10T05:19:23.104ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
34032025-01-10T05:19:23.104ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
34042025-01-10T05:19:23.104ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
34052025-01-10T05:19:23.104ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
34062025-01-10T05:19:23.104ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08
34072025-01-10T05:19:23.104ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
34082025-01-10T05:19:23.105ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
34092025-01-10T05:19:23.107ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
34102025-01-10T05:19:23.107ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "35f269d2-9d8d-46d0-9a5b-ed3cbd73dfe4", "content-length": "2", "date": "Fri, 10 Jan 2025 05:19:23 GMT"} })
34112025-01-10T05:19:23.107ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
34122025-01-10T05:19:23.107ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b2e3609-2de5-48de-99f9-159c2bf24f4f", "content-length": "52", "date": "Fri, 10 Jan 2025 05:19:23 GMT"} })
34132025-01-10T05:19:23.107ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 switch_location = Switch0
34142025-01-10T05:19:23.108ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
34152025-01-10T05:19:23.204ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "fdbfedce-827e-4980-b24c-d1bdf0377fe2", "date": "Fri, 10 Jan 2025 05:19:22 GMT"} })
34162025-01-10T05:19:23.204ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 switch = Switch0
34172025-01-10T05:19:23.204ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
34182025-01-10T05:19:23.204ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff8619a5-9e46-4262-8b8f-06494c3d8391", "content-length": "682", "date": "Fri, 10 Jan 2025 05:19:23 GMT"} })
34192025-01-10T05:19:23.205ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
34202025-01-10T05:19:23.205ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
34212025-01-10T05:19:23.208ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
34222025-01-10T05:19:23.208ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
34232025-01-10T05:19:23.211ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
34242025-01-10T05:19:23.211ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
34252025-01-10T05:19:23.213ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
34262025-01-10T05:19:23.213ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
34272025-01-10T05:19:23.220ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 566.02368ms iteration = 28 status = {}
34282025-01-10T05:19:23.641ZDEBGnexus (producer-registration-task): registering / renewing oximeter producer lease with Nexus
34292025-01-10T05:19:23.641ZDEBGnexus (producer-registration-task): using nexus address for registration addr = [fd00:1122:3344:101::c]:12221
34302025-01-10T05:19:23.671ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::c]:12221/metrics/producers
34312025-01-10T05:19:23.671ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:1023 local_addr = [fd00:1122:3344:101::c]:12221 remote_addr = [fd00:1122:3344:101::c]:52649
34322025-01-10T05:19:23.672ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::c]:52649 req_id = 02d7eb90-1bce-44e6-aba2-e6e10ed6c37c roles = RoleSet { roles: {} } uri = /metrics/producers
34332025-01-10T05:19:23.672ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::c]:52649 req_id = 02d7eb90-1bce-44e6-aba2-e6e10ed6c37c resource = Database result = Ok(()) uri = /metrics/producers
34342025-01-10T05:19:23.707ZINFOnexus: assigned collector to new producer collector_id = b6508fc7-5132-4a68-971d-9bf335b84514 file = nexus/src/app/oximeter.rs:76 producer_id = 3ca89dd1-7ada-4fe6-8600-a84d41dfa9d8
34352025-01-10T05:19:23.707ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 35173 local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::c]:52649 req_id = 02d7eb90-1bce-44e6-aba2-e6e10ed6c37c response_code = 201 uri = /metrics/producers
34362025-01-10T05:19:23.707ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "02d7eb90-1bce-44e6-aba2-e6e10ed6c37c", "content-length": "41", "date": "Fri, 10 Jan 2025 05:19:23 GMT"} })
34372025-01-10T05:19:23.709ZDEBGnexus (producer-registration-task): registered with nexus successfully
34382025-01-10T05:19:23.709ZDEBGnexus (producer-registration-task): pausing until time to renew lease lease_duration = 600s wait_period = 150s
34392025-01-10T05:19:27.546ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 322 local_addr = [fd00:1122:3344:101::c]:57161 method = GET remote_addr = [fd00:1122:3344:101::d]:46894 req_id = 27d752d8-4e4f-4be7-834c-994f1bcc9e72 response_code = 200 uri = /3ca89dd1-7ada-4fe6-8600-a84d41dfa9d8
34402025-01-10T05:19:32.653ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 83 reason = Timeout
34412025-01-10T05:19:32.653ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 83 reason = Timeout
34422025-01-10T05:19:32.653ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
34432025-01-10T05:19:32.653ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
34442025-01-10T05:19:32.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
34452025-01-10T05:19:32.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
34462025-01-10T05:19:32.656ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
34472025-01-10T05:19:32.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
34482025-01-10T05:19:32.658ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.556198ms iteration = 83 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
34492025-01-10T05:19:32.682ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34502025-01-10T05:19:32.685ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
34512025-01-10T05:19:32.685ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
34522025-01-10T05:19:32.685ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
34532025-01-10T05:19:32.688ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
34542025-01-10T05:19:32.688ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
34552025-01-10T05:19:32.722ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34562025-01-10T05:19:32.725ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 43ea24da-5730-4a85-9a8e-76459f9b69b1, lookup_type: ById(43ea24da-5730-4a85-9a8e-76459f9b69b1) } result = Ok(())
34572025-01-10T05:19:32.725ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
34582025-01-10T05:19:32.725ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
34592025-01-10T05:19:32.733ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 original_time_created = 2025-01-10 05:05:43.319091 UTC target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 time_created = 2025-01-10 05:05:43.319091 UTC
34602025-01-10T05:19:32.733ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 79.624654ms iteration = 83 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"43ea24da-5730-4a85-9a8e-76459f9b69b1","time_created":"2025-01-10T05:05:43.319091Z"}
34612025-01-10T05:19:37.546ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 320 local_addr = [fd00:1122:3344:101::c]:57161 method = GET remote_addr = [fd00:1122:3344:101::d]:46894 req_id = 2fac4dbd-29b5-4d6a-939c-87bd4daff475 response_code = 200 uri = /3ca89dd1-7ada-4fe6-8600-a84d41dfa9d8
34622025-01-10T05:19:42.653ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 84 reason = Timeout
34632025-01-10T05:19:42.653ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 84 reason = Timeout
34642025-01-10T05:19:42.653ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
34652025-01-10T05:19:42.653ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
34662025-01-10T05:19:42.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
34672025-01-10T05:19:42.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
34682025-01-10T05:19:42.656ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34692025-01-10T05:19:42.656ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
34702025-01-10T05:19:42.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
34712025-01-10T05:19:42.658ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
34722025-01-10T05:19:42.658ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
34732025-01-10T05:19:42.659ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
34742025-01-10T05:19:42.661ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
34752025-01-10T05:19:42.661ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
34762025-01-10T05:19:42.662ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34772025-01-10T05:19:42.665ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 43ea24da-5730-4a85-9a8e-76459f9b69b1, lookup_type: ById(43ea24da-5730-4a85-9a8e-76459f9b69b1) } result = Ok(())
34782025-01-10T05:19:42.665ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
34792025-01-10T05:19:42.665ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
34802025-01-10T05:19:42.677ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 original_time_created = 2025-01-10 05:05:43.319091 UTC target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 time_created = 2025-01-10 05:05:43.319091 UTC
34812025-01-10T05:19:42.677ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 24.062071ms iteration = 84 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"43ea24da-5730-4a85-9a8e-76459f9b69b1","time_created":"2025-01-10T05:05:43.319091Z"}
34822025-01-10T05:19:42.685ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 31.73306ms iteration = 84 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
34832025-01-10T05:19:47.546ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 325 local_addr = [fd00:1122:3344:101::c]:57161 method = GET remote_addr = [fd00:1122:3344:101::d]:46894 req_id = 109e7b72-3921-4f82-949e-3dde54640b35 response_code = 200 uri = /3ca89dd1-7ada-4fe6-8600-a84d41dfa9d8
34842025-01-10T05:19:52.653ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 85 reason = Timeout
34852025-01-10T05:19:52.653ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 29 reason = Timeout
34862025-01-10T05:19:52.653ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
34872025-01-10T05:19:52.653ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
34882025-01-10T05:19:52.653ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 15 reason = Timeout
34892025-01-10T05:19:52.653ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
34902025-01-10T05:19:52.653ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 15 reason = Timeout
34912025-01-10T05:19:52.653ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 29 reason = Timeout
34922025-01-10T05:19:52.653ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
34932025-01-10T05:19:52.653ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 15 reason = Timeout
34942025-01-10T05:19:52.653ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
34952025-01-10T05:19:52.653ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
34962025-01-10T05:19:52.653ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
34972025-01-10T05:19:52.653ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 85 reason = Timeout
34982025-01-10T05:19:52.653ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
34992025-01-10T05:19:52.653ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 29 reason = Timeout
35002025-01-10T05:19:52.653ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
35012025-01-10T05:19:52.653ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 29 reason = Timeout
35022025-01-10T05:19:52.653ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 29 reason = Timeout
35032025-01-10T05:19:52.653ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
35042025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 29 reason = Timeout
35052025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 29 reason = Timeout
35062025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 15 reason = Timeout
35072025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
35082025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
35092025-01-10T05:19:52.654ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-01-10 05:09:52.653978390 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
35102025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
35112025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 15 reason = Timeout
35122025-01-10T05:19:52.654ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
35132025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 29 reason = Timeout
35142025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
35152025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
35162025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 29 reason = Timeout
35172025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
35182025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
35192025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 29 reason = Timeout
35202025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
35212025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 29 reason = Timeout
35222025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
35232025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 15 reason = Timeout
35242025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 31 reason = Timeout
35252025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
35262025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 15 reason = Timeout
35272025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 17 reason = Timeout
35282025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 29 reason = Timeout
35292025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
35302025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 15 reason = Timeout
35312025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
35322025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
35332025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 15 reason = Timeout
35342025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
35352025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
35362025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 15 reason = Timeout
35372025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
35382025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 16 reason = Timeout
35392025-01-10T05:19:52.654ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:83 reason = blueprint disabled target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1
35402025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 17 reason = Timeout
35412025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 29 reason = Timeout
35422025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 14.21\u{b5}s iteration = 16 status = {"enabled":false,"target_id":"43ea24da-5730-4a85-9a8e-76459f9b69b1"}
35432025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
35442025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
35452025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
35462025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
35472025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
35482025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
35492025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
35502025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
35512025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
35522025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
35532025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
35542025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
35552025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
35562025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
35572025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
35582025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
35592025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
35602025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
35612025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
35622025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
35632025-01-10T05:19:52.654ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("30a043aa-79b8-40f0-889b-b1aa4686508a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("95b462f2-3c97-4463-9c08-5404518f33d2.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("95b462f2-3c97-4463-9c08-5404518f33d2.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 27203, tv_nsec: 164200103 } })
35642025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
35652025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
35662025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
35672025-01-10T05:19:52.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
35682025-01-10T05:19:52.654ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("53514ca5-28a5-4b76-afd8-362873acf1d4.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("b3737e73-ceea-4b83-966e-05b55558955a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("e93887bd-ffe8-46b8-ad6f-92e0665448ab.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("e93887bd-ffe8-46b8-ad6f-92e0665448ab.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 27203, tv_nsec: 164315997 } })
35692025-01-10T05:19:52.655ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
35702025-01-10T05:19:52.655ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.002662ms iteration = 15 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
35712025-01-10T05:19:52.655ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 29 reason = Timeout
35722025-01-10T05:19:52.655ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
35732025-01-10T05:19:52.655ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
35742025-01-10T05:19:52.655ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
35752025-01-10T05:19:52.655ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.640633ms iteration = 15 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353"]}
35762025-01-10T05:19:52.657ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35772025-01-10T05:19:52.657ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35782025-01-10T05:19:52.657ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.539965ms iteration = 29 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
35792025-01-10T05:19:52.658ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35802025-01-10T05:19:52.658ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 27203, tv_nsec: 168178872 } })
35812025-01-10T05:19:52.658ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1103
35822025-01-10T05:19:52.661ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
35832025-01-10T05:19:52.661ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
35842025-01-10T05:19:52.661ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
35852025-01-10T05:19:52.661ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
35862025-01-10T05:19:52.661ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
35872025-01-10T05:19:52.661ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
35882025-01-10T05:19:52.661ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
35892025-01-10T05:19:52.661ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
35902025-01-10T05:19:52.661ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
35912025-01-10T05:19:52.663ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 27203, tv_nsec: 172693849 } })
35922025-01-10T05:19:52.663ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1103
35932025-01-10T05:19:52.663ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 10.113118ms iteration = 15 status = {"errors":[],"found_port_ok":[]}
35942025-01-10T05:19:52.665ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35952025-01-10T05:19:52.666ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-01-10 05:09:52.653978390 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
35962025-01-10T05:19:52.666ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 12.354701ms iteration = 15 status = {"expiration":"2025-01-10T05:09:52.653978390Z","pruned":[]}
35972025-01-10T05:19:52.666ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35982025-01-10T05:19:52.667ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
35992025-01-10T05:19:52.667ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
36002025-01-10T05:19:52.667ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
36012025-01-10T05:19:52.667ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
36022025-01-10T05:19:52.667ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
36032025-01-10T05:19:52.667ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
36042025-01-10T05:19:52.667ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
36052025-01-10T05:19:52.668ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
36062025-01-10T05:19:52.668ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
36072025-01-10T05:19:52.668ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
36082025-01-10T05:19:52.670ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
36092025-01-10T05:19:52.671ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
36102025-01-10T05:19:52.672ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
36112025-01-10T05:19:52.673ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
36122025-01-10T05:19:52.674ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
36132025-01-10T05:19:52.674ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
36142025-01-10T05:19:52.675ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
36152025-01-10T05:19:52.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
36162025-01-10T05:19:52.679ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36172025-01-10T05:19:52.682ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 27203, tv_nsec: 191816060 } })
36182025-01-10T05:19:52.682ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
36192025-01-10T05:19:52.682ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1103 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08
36202025-01-10T05:19:52.690ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36212025-01-10T05:19:52.692ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
36222025-01-10T05:19:52.692ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
36232025-01-10T05:19:52.692ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
36242025-01-10T05:19:52.694ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36252025-01-10T05:19:52.697ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
36262025-01-10T05:19:52.697ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
36272025-01-10T05:19:52.697ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
36282025-01-10T05:19:52.700ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
36292025-01-10T05:19:52.700ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36302025-01-10T05:19:52.700ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
36312025-01-10T05:19:52.700ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
36322025-01-10T05:19:52.700ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
36332025-01-10T05:19:52.700ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
36342025-01-10T05:19:52.700ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
36352025-01-10T05:19:52.702ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36362025-01-10T05:19:52.704ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
36372025-01-10T05:19:52.705ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
36382025-01-10T05:19:52.705ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 50.405456ms iteration = 29 status = {"errors":[],"finish_invoked_ok":[]}
36392025-01-10T05:19:52.705ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 43ea24da-5730-4a85-9a8e-76459f9b69b1, lookup_type: ById(43ea24da-5730-4a85-9a8e-76459f9b69b1) } result = Ok(())
36402025-01-10T05:19:52.705ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
36412025-01-10T05:19:52.705ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
36422025-01-10T05:19:52.706ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 52.146152ms iteration = 29 status = {"errors":[],"garbage_collect_requested":[]}
36432025-01-10T05:19:52.706ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
36442025-01-10T05:19:52.706ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
36452025-01-10T05:19:52.706ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
36462025-01-10T05:19:52.707ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
36472025-01-10T05:19:52.707ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:171 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
36482025-01-10T05:19:52.707ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 53.083992ms iteration = 15 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
36492025-01-10T05:19:52.711ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
36502025-01-10T05:19:52.711ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36512025-01-10T05:19:52.711ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
36522025-01-10T05:19:52.712ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
36532025-01-10T05:19:52.713ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
36542025-01-10T05:19:52.714ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
36552025-01-10T05:19:52.714ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
36562025-01-10T05:19:52.714ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
36572025-01-10T05:19:52.718ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36582025-01-10T05:19:52.719ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36592025-01-10T05:19:52.719ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
36602025-01-10T05:19:52.719ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
36612025-01-10T05:19:52.719ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
36622025-01-10T05:19:52.719ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
36632025-01-10T05:19:52.722ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
36642025-01-10T05:19:52.722ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
36652025-01-10T05:19:52.722ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
36662025-01-10T05:19:52.724ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
36672025-01-10T05:19:52.726ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36682025-01-10T05:19:52.726ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36692025-01-10T05:19:52.729ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
36702025-01-10T05:19:52.729ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
36712025-01-10T05:19:52.729ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 75.584463ms iteration = 29 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
36722025-01-10T05:19:52.729ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
36732025-01-10T05:19:52.731ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59, lookup_type: ById(a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59) } result = Ok(())
36742025-01-10T05:19:52.731ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
36752025-01-10T05:19:52.731ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
36762025-01-10T05:19:52.731ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
36772025-01-10T05:19:52.731ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
36782025-01-10T05:19:52.731ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
36792025-01-10T05:19:52.731ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
36802025-01-10T05:19:52.731ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
36812025-01-10T05:19:52.731ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
36822025-01-10T05:19:52.731ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 77.971301ms iteration = 15 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
36832025-01-10T05:19:52.732ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
36842025-01-10T05:19:52.733ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36852025-01-10T05:19:52.733ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 80.013977ms iteration = 29 status = 0
36862025-01-10T05:19:52.736ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 original_time_created = 2025-01-10 05:05:43.319091 UTC target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 time_created = 2025-01-10 05:05:43.319091 UTC
36872025-01-10T05:19:52.736ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 82.725434ms iteration = 85 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"43ea24da-5730-4a85-9a8e-76459f9b69b1","time_created":"2025-01-10T05:05:43.319091Z"}
36882025-01-10T05:19:52.738ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
36892025-01-10T05:19:52.738ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
36902025-01-10T05:19:52.738ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
36912025-01-10T05:19:52.746ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 92.786581ms iteration = 85 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
36922025-01-10T05:19:52.748ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
36932025-01-10T05:19:52.748ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
36942025-01-10T05:19:52.750ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
36952025-01-10T05:19:52.750ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
36962025-01-10T05:19:52.753ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
36972025-01-10T05:19:52.753ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
36982025-01-10T05:19:52.757ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
36992025-01-10T05:19:52.757ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
37002025-01-10T05:19:52.757ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
37012025-01-10T05:19:52.757ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
37022025-01-10T05:19:52.765ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
37032025-01-10T05:19:52.765ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
37042025-01-10T05:19:52.765ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
37052025-01-10T05:19:52.765ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
37062025-01-10T05:19:52.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
37072025-01-10T05:19:52.769ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37082025-01-10T05:19:52.772ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
37092025-01-10T05:19:52.772ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 118.101533ms iteration = 29 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
37102025-01-10T05:19:52.772ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
37112025-01-10T05:19:52.772ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
37122025-01-10T05:19:52.772ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
37132025-01-10T05:19:52.775ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 121.267086ms iteration = 29 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
37142025-01-10T05:19:52.787ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
37152025-01-10T05:19:52.791ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37162025-01-10T05:19:52.792ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1111 zone_address = fd00:1122:3344:101::2
37172025-01-10T05:19:52.792ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
37182025-01-10T05:19:52.792ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb3fe78d-e94a-4221-ac3a-e1b4eae331d2", "content-length": "26", "date": "Fri, 10 Jan 2025 05:19:52 GMT"} })
37192025-01-10T05:19:52.792ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1116 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
37202025-01-10T05:19:52.792ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 138.34728ms iteration = 29 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
37212025-01-10T05:19:52.792ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
37222025-01-10T05:19:52.792ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = bca3dac0-9831-415b-86b5-c6c6cc12ed56 dns_zone_name = oxide.test found_so_far = 0 version = 2
37232025-01-10T05:19:52.792ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
37242025-01-10T05:19:52.793ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
37252025-01-10T05:19:52.793ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
37262025-01-10T05:19:52.793ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
37272025-01-10T05:19:52.793ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
37282025-01-10T05:19:52.793ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
37292025-01-10T05:19:52.794ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1148 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
37302025-01-10T05:19:52.794ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37312025-01-10T05:19:52.797ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
37322025-01-10T05:19:52.797ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
37332025-01-10T05:19:52.797ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 143.144726ms iteration = 29 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
37342025-01-10T05:19:52.797ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
37352025-01-10T05:19:52.797ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
37362025-01-10T05:19:52.797ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
37372025-01-10T05:19:52.799ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = bca3dac0-9831-415b-86b5-c6c6cc12ed56 dns_zone_name = oxide.test found_so_far = 1 version = 2
37382025-01-10T05:19:52.799ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
37392025-01-10T05:19:52.799ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-01-10 05:06:09.259013 UTC dns_group = external generation = 2
37402025-01-10T05:19:52.799ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 144.900042ms iteration = 15 status = {"generation":2}
37412025-01-10T05:19:52.803ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
37422025-01-10T05:19:52.803ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
37432025-01-10T05:19:52.810ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
37442025-01-10T05:19:52.810ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 70f09b61-dde8-4da2-af2f-06138d5a975e dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
37452025-01-10T05:19:52.810ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
37462025-01-10T05:19:52.810ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
37472025-01-10T05:19:52.811ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37482025-01-10T05:19:52.815ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
37492025-01-10T05:19:52.815ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
37502025-01-10T05:19:52.815ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
37512025-01-10T05:19:52.817ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 70f09b61-dde8-4da2-af2f-06138d5a975e dns_zone_name = control-plane.oxide.internal found_so_far = 51 version = 1
37522025-01-10T05:19:52.817ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
37532025-01-10T05:19:52.817ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-01-10 05:06:08.425967 UTC dns_group = internal generation = 1
37542025-01-10T05:19:52.817ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 163.324151ms iteration = 15 status = {"generation":1}
37552025-01-10T05:19:52.818ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1111 zone_address = fd00:1122:3344:101::2
37562025-01-10T05:19:52.818ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
37572025-01-10T05:19:52.818ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "b5d2a6a8-f29e-4855-b11e-caebcbb2aa42", "content-length": "26", "date": "Fri, 10 Jan 2025 05:19:52 GMT"} })
37582025-01-10T05:19:52.818ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1116 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
37592025-01-10T05:19:52.819ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1148 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
37602025-01-10T05:19:52.820ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
37612025-01-10T05:19:52.820ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 166.125741ms iteration = 15 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
37622025-01-10T05:19:52.832ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 178.39723ms iteration = 31 status = {"physical_disks_added":0}
37632025-01-10T05:19:52.838ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1111 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 zone_address = fd00:1122:3344:101::2
37642025-01-10T05:19:52.838ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
37652025-01-10T05:19:52.838ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "7458dc79-d08a-4022-9a84-d43fffd57d25", "content-length": "26", "date": "Fri, 10 Jan 2025 05:19:52 GMT"} })
37662025-01-10T05:19:52.838ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1116 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
37672025-01-10T05:19:52.840ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1148 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 79fde928-482f-41ab-aee3-3f2478145f08
37682025-01-10T05:19:52.845ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-01-10T05:05:45.360170Z, time_modified: 2025-01-10T05:05:45.360170Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 48a76ac8-2f94-4847-be0f-1cb4bfb661aa, db_silo: Silo { identity: SiloIdentity { id: 48a76ac8-2f94-4847-be0f-1cb4bfb661aa, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-01-10T05:06:09.192249Z, time_modified: 2025-01-10T05:06:09.192249Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "0de51367feb2d6f1959542dbb73fb728546b6d8dd71a0ec2fd5083120a87d20f" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 48a76ac8-2f94-4847-be0f-1cb4bfb661aa, db_silo: Silo { identity: SiloIdentity { id: 48a76ac8-2f94-4847-be0f-1cb4bfb661aa, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-01-10T05:06:09.192249Z, time_modified: 2025-01-10T05:06:09.192249Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "0de51367feb2d6f1959542dbb73fb728546b6d8dd71a0ec2fd5083120a87d20f" }] }) }
37692025-01-10T05:19:52.845ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 191.627939ms iteration = 15 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"48a76ac8-2f94-4847-be0f-1cb4bfb661aa","tls_certs":["0de51367feb2d6f1959542dbb73fb728546b6d8dd71a0ec2fd5083120a87d20f"]}},"default_endpoint":{"silo_id":"48a76ac8-2f94-4847-be0f-1cb4bfb661aa","tls_certs":["0de51367feb2d6f1959542dbb73fb728546b6d8dd71a0ec2fd5083120a87d20f"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
37702025-01-10T05:19:52.939ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
37712025-01-10T05:19:52.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
37722025-01-10T05:19:52.941ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37732025-01-10T05:19:52.943ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
37742025-01-10T05:19:52.943ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
37752025-01-10T05:19:52.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
37762025-01-10T05:19:52.960ZINFOnexus (BackgroundTasks): VPC route manager sled a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
37772025-01-10T05:19:52.960ZDEBGnexus (BackgroundTasks): client request SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
37782025-01-10T05:19:52.960ZDEBGnexus (BackgroundTasks): client response SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "23770cdf-d570-40db-9b5d-87a04fb8dcd8", "content-length": "372", "date": "Fri, 10 Jan 2025 05:19:52 GMT"} })
37792025-01-10T05:19:52.960ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
37802025-01-10T05:19:52.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
37812025-01-10T05:19:52.962ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37822025-01-10T05:19:52.964ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
37832025-01-10T05:19:52.964ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
37842025-01-10T05:19:52.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
37852025-01-10T05:19:52.969ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
37862025-01-10T05:19:52.969ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "d9612253-625d-400a-8c2d-26e567648676", "date": "Fri, 10 Jan 2025 05:19:52 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
37872025-01-10T05:19:52.969ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
37882025-01-10T05:19:52.971ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
37892025-01-10T05:19:52.972ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "5f8fce9d-7ca0-41c7-9ff3-d20bcf433d30", "date": "Fri, 10 Jan 2025 05:19:52 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
37902025-01-10T05:19:52.973ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
37912025-01-10T05:19:52.974ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 320.531173ms iteration = 29 status = {}
37922025-01-10T05:19:52.975ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
37932025-01-10T05:19:52.975ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 321.492093ms iteration = 17 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
37942025-01-10T05:19:52.998ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
37952025-01-10T05:19:52.999ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "d0c47344-d8d4-4aca-8672-4c43e0b0bc42", "content-length": "2", "date": "Fri, 10 Jan 2025 05:19:52 GMT"} })
37962025-01-10T05:19:53.000ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
37972025-01-10T05:19:53.000ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
37982025-01-10T05:19:53.002ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 348.689006ms iteration = 29 status = null
37992025-01-10T05:19:53.002ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
38002025-01-10T05:19:53.002ZDEBGnexus (BackgroundTasks): client request SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
38012025-01-10T05:19:53.003ZDEBGnexus (BackgroundTasks): client response SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "ee8dcb73-e723-46ac-8e1d-fcaa57f4f90c", "date": "Fri, 10 Jan 2025 05:19:52 GMT"} })
38022025-01-10T05:19:53.003ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
38032025-01-10T05:19:53.003ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
38042025-01-10T05:19:53.005ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
38052025-01-10T05:19:53.006ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
38062025-01-10T05:19:53.008ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
38072025-01-10T05:19:53.009ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
38082025-01-10T05:19:53.012ZINFOnexus (BackgroundTasks): VPC route manager sled a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
38092025-01-10T05:19:53.012ZDEBGnexus (BackgroundTasks): client request SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
38102025-01-10T05:19:53.013ZDEBGnexus (BackgroundTasks): client response SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "c1eec16c-a68f-4fd1-8dd6-027ac27de617", "date": "Fri, 10 Jan 2025 05:19:52 GMT"} })
38112025-01-10T05:19:53.014ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 360.688676ms iteration = 29 status = {}
38122025-01-10T05:19:53.052ZDEBGnexus (BackgroundTasks): client request SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
38132025-01-10T05:19:53.052ZDEBGnexus (BackgroundTasks): client response SledAgent = a85eeb1a-bbdd-4901-9cfb-db76e1cf9b59 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "922b0a07-b386-4164-9428-7fe0f8258151", "content-length": "691", "date": "Fri, 10 Jan 2025 05:19:52 GMT"} })
38142025-01-10T05:19:53.052ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
38152025-01-10T05:19:53.052ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
38162025-01-10T05:19:53.052ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
38172025-01-10T05:19:53.053ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
38182025-01-10T05:19:53.053ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "ed8664ff-4685-4273-a6ab-ce14af4a894c", "date": "Fri, 10 Jan 2025 05:19:52 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
38192025-01-10T05:19:53.053ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
38202025-01-10T05:19:53.055ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 401.51088ms iteration = 29 status = {}
38212025-01-10T05:19:53.055ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "52a86b5c-5cdf-4217-b751-5d9e7f196d85", "date": "Fri, 10 Jan 2025 05:19:53 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
38222025-01-10T05:19:53.055ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
38232025-01-10T05:19:53.058ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "11e5f05e-bbe2-4a2e-8514-2f1287c02168", "date": "Fri, 10 Jan 2025 05:19:53 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
38242025-01-10T05:19:53.058ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
38252025-01-10T05:19:53.059ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
38262025-01-10T05:19:53.059ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 405.085256ms iteration = 17 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
38272025-01-10T05:19:53.103ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
38282025-01-10T05:19:53.104ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
38292025-01-10T05:19:53.106ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 5263359f-a80d-45e0-9595-0b15b904e78f, rack_id: 79fde928-482f-41ab-aee3-3f2478145f08, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(77858eb2-1069-4750-be5e-f3dafe20063a) } rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 switch_location = Switch0
38302025-01-10T05:19:53.106ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
38312025-01-10T05:19:53.106ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
38322025-01-10T05:19:53.118ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
38332025-01-10T05:19:53.119ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf123542-be5c-4746-b409-77d1607f6402", "content-length": "134", "date": "Fri, 10 Jan 2025 05:19:52 GMT"} })
38342025-01-10T05:19:53.119ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
38352025-01-10T05:19:53.119ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 77858eb2-1069-4750-be5e-f3dafe20063a
38362025-01-10T05:19:53.119ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
38372025-01-10T05:19:53.119ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
38382025-01-10T05:19:53.119ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
38392025-01-10T05:19:53.119ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
38402025-01-10T05:19:53.119ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
38412025-01-10T05:19:53.119ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
38422025-01-10T05:19:53.119ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
38432025-01-10T05:19:53.119ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
38442025-01-10T05:19:53.120ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08
38452025-01-10T05:19:53.120ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
38462025-01-10T05:19:53.120ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
38472025-01-10T05:19:53.122ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
38482025-01-10T05:19:53.125ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "575a4e7c-a8ce-4df1-a736-09145a346a0d", "content-length": "2", "date": "Fri, 10 Jan 2025 05:19:52 GMT"} })
38492025-01-10T05:19:53.125ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
38502025-01-10T05:19:53.125ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc5f9dfe-f194-411f-8b6e-89fbc8ddfcd2", "content-length": "52", "date": "Fri, 10 Jan 2025 05:19:52 GMT"} })
38512025-01-10T05:19:53.125ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 switch_location = Switch0
38522025-01-10T05:19:53.125ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
38532025-01-10T05:19:53.226ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "84f6682f-1027-4940-943d-0280dd34c63a", "date": "Fri, 10 Jan 2025 05:19:52 GMT"} })
38542025-01-10T05:19:53.226ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 switch = Switch0
38552025-01-10T05:19:53.226ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
38562025-01-10T05:19:53.226ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 79fde928-482f-41ab-aee3-3f2478145f08 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "43856c6d-a3fd-4ebf-9b93-052da2d24d1a", "content-length": "682", "date": "Fri, 10 Jan 2025 05:19:52 GMT"} })
38572025-01-10T05:19:53.226ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
38582025-01-10T05:19:53.227ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
38592025-01-10T05:19:53.229ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
38602025-01-10T05:19:53.229ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
38612025-01-10T05:19:53.232ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
38622025-01-10T05:19:53.232ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
38632025-01-10T05:19:53.234ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
38642025-01-10T05:19:53.234ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
38652025-01-10T05:19:53.242ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 588.169228ms iteration = 29 status = {}
38662025-01-10T05:19:57.546ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.13.0/src/server.rs:863 latency_us = 288 local_addr = [fd00:1122:3344:101::c]:57161 method = GET remote_addr = [fd00:1122:3344:101::d]:46894 req_id = 7fb42a81-e6ad-47ba-97cf-f0f4a12c2a04 response_code = 200 uri = /3ca89dd1-7ada-4fe6-8600-a84d41dfa9d8
38672025-01-10T05:20:02.653ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 86 reason = Timeout
38682025-01-10T05:20:02.653ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
38692025-01-10T05:20:02.653ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 86 reason = Timeout
38702025-01-10T05:20:02.653ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
38712025-01-10T05:20:02.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
38722025-01-10T05:20:02.654ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
38732025-01-10T05:20:02.656ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38742025-01-10T05:20:02.659ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
38752025-01-10T05:20:02.659ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
38762025-01-10T05:20:02.659ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
38772025-01-10T05:20:02.661ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
38782025-01-10T05:20:02.661ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
38792025-01-10T05:20:02.663ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38802025-01-10T05:20:02.666ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 43ea24da-5730-4a85-9a8e-76459f9b69b1, lookup_type: ById(43ea24da-5730-4a85-9a8e-76459f9b69b1) } result = Ok(())
38812025-01-10T05:20:02.666ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
38822025-01-10T05:20:02.666ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
38832025-01-10T05:20:02.678ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 original_time_created = 2025-01-10 05:05:43.319091 UTC target_id = 43ea24da-5730-4a85-9a8e-76459f9b69b1 time_created = 2025-01-10 05:05:43.319091 UTC
38842025-01-10T05:20:02.678ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 25.114505ms iteration = 86 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"43ea24da-5730-4a85-9a8e-76459f9b69b1","time_created":"2025-01-10T05:05:43.319091Z"}
38852025-01-10T05:20:02.682ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
38862025-01-10T05:20:02.682ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
38872025-01-10T05:20:02.709ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 55.881453ms iteration = 86 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}