Max level shown:
12025-02-25T04:00:03.145ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 69 reason = Timeout
22025-02-25T04:00:03.145ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
32025-02-25T04:00:03.145ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 69 reason = Timeout
42025-02-25T04:00:03.145ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
52025-02-25T04:00:03.145ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
62025-02-25T04:00:03.145ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
72025-02-25T04:00:03.147ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
82025-02-25T04:00:03.147ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
92025-02-25T04:00:03.148ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102025-02-25T04:00:03.150ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
112025-02-25T04:00:03.150ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
122025-02-25T04:00:03.150ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
132025-02-25T04:00:03.152ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
142025-02-25T04:00:03.153ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
152025-02-25T04:00:03.154ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
162025-02-25T04:00:03.157ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 84829298-e6df-4d09-ba9a-550ee8541f9e, lookup_type: ById(84829298-e6df-4d09-ba9a-550ee8541f9e) } result = Ok(())
172025-02-25T04:00:03.157ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
182025-02-25T04:00:03.157ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
192025-02-25T04:00:03.171ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 84829298-e6df-4d09-ba9a-550ee8541f9e original_time_created = 2025-02-25 03:48:33.610641 UTC target_id = 84829298-e6df-4d09-ba9a-550ee8541f9e time_created = 2025-02-25 03:48:33.610641 UTC
202025-02-25T04:00:03.171ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 25.662794ms iteration = 69 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"84829298-e6df-4d09-ba9a-550ee8541f9e","time_created":"2025-02-25T03:48:33.610641Z"}
212025-02-25T04:00:03.184ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 39.223567ms iteration = 69 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
222025-02-25T04:00:05.635ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.15.1/src/server.rs:867 latency_us = 339 local_addr = [fd00:1122:3344:101::c]:52384 method = GET remote_addr = [fd00:1122:3344:101::d]:48316 req_id = aeca59c3-22de-43c4-ab0a-410db2c6be02 response_code = 200 uri = /d8b217e1-a0a4-419c-82c9-30f4d54f8fbd
232025-02-25T04:00:10.626ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56550 req_id = c7d4d83c-645a-4745-ba97-e9fa61fac831 roles = RoleSet { roles: {} } uri = /metrics/collectors/3cd5b255-a8f1-4eea-aa4f-9c536fb3dad1/producers?sort_by=id_ascending
242025-02-25T04:00:10.626ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56550 req_id = c7d4d83c-645a-4745-ba97-e9fa61fac831 resource = Database result = Ok(()) uri = /metrics/collectors/3cd5b255-a8f1-4eea-aa4f-9c536fb3dad1/producers?sort_by=id_ascending
252025-02-25T04:00:10.629ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.15.1/src/server.rs:867 latency_us = 3574 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56550 req_id = c7d4d83c-645a-4745-ba97-e9fa61fac831 response_code = 200 uri = /metrics/collectors/3cd5b255-a8f1-4eea-aa4f-9c536fb3dad1/producers?sort_by=id_ascending
262025-02-25T04:00:10.629ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56550 req_id = 35edd28b-f5c8-4022-b23b-58bf486d87ac roles = RoleSet { roles: {} } uri = /metrics/collectors/3cd5b255-a8f1-4eea-aa4f-9c536fb3dad1/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmMjc5Y2EyMS0xNDdlLTQwNTAtODVmZS0zYTYzMTIxYjM0M2YifX0%3D
272025-02-25T04:00:10.630ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56550 req_id = 35edd28b-f5c8-4022-b23b-58bf486d87ac resource = Database result = Ok(()) uri = /metrics/collectors/3cd5b255-a8f1-4eea-aa4f-9c536fb3dad1/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmMjc5Y2EyMS0xNDdlLTQwNTAtODVmZS0zYTYzMTIxYjM0M2YifX0%3D
282025-02-25T04:00:10.632ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.15.1/src/server.rs:867 latency_us = 2600 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:56550 req_id = 35edd28b-f5c8-4022-b23b-58bf486d87ac response_code = 200 uri = /metrics/collectors/3cd5b255-a8f1-4eea-aa4f-9c536fb3dad1/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmMjc5Y2EyMS0xNDdlLTQwNTAtODVmZS0zYTYzMTIxYjM0M2YifX0%3D
292025-02-25T04:00:13.145ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 24 reason = Timeout
302025-02-25T04:00:13.145ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
312025-02-25T04:00:13.145ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 24 reason = Timeout
322025-02-25T04:00:13.145ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 24 reason = Timeout
332025-02-25T04:00:13.145ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
342025-02-25T04:00:13.145ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
352025-02-25T04:00:13.145ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 24 reason = Timeout
362025-02-25T04:00:13.145ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
372025-02-25T04:00:13.145ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 70 reason = Timeout
382025-02-25T04:00:13.145ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 24 reason = Timeout
392025-02-25T04:00:13.145ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
402025-02-25T04:00:13.145ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
412025-02-25T04:00:13.145ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
422025-02-25T04:00:13.145ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 70 reason = Timeout
432025-02-25T04:00:13.145ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 24 reason = Timeout
442025-02-25T04:00:13.145ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
452025-02-25T04:00:13.145ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
462025-02-25T04:00:13.145ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 24 reason = Timeout
472025-02-25T04:00:13.145ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 24 reason = Timeout
482025-02-25T04:00:13.145ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
492025-02-25T04:00:13.145ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
502025-02-25T04:00:13.145ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 24 reason = Timeout
512025-02-25T04:00:13.145ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
522025-02-25T04:00:13.145ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 24 reason = Timeout
532025-02-25T04:00:13.145ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
542025-02-25T04:00:13.145ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 24 reason = Timeout
552025-02-25T04:00:13.145ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
562025-02-25T04:00:13.145ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 24 reason = Timeout
572025-02-25T04:00:13.145ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
582025-02-25T04:00:13.145ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 24 reason = Timeout
592025-02-25T04:00:13.145ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
602025-02-25T04:00:13.145ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 26 reason = Timeout
612025-02-25T04:00:13.145ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
622025-02-25T04:00:13.145ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 24 reason = Timeout
632025-02-25T04:00:13.145ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
642025-02-25T04:00:13.145ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 24 reason = Timeout
652025-02-25T04:00:13.145ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
662025-02-25T04:00:13.145ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 24 reason = Timeout
672025-02-25T04:00:13.145ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
682025-02-25T04:00:13.145ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
692025-02-25T04:00:13.145ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
702025-02-25T04:00:13.145ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
712025-02-25T04:00:13.145ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
722025-02-25T04:00:13.145ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
732025-02-25T04:00:13.145ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
742025-02-25T04:00:13.145ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
752025-02-25T04:00:13.145ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
762025-02-25T04:00:13.145ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
772025-02-25T04:00:13.145ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
782025-02-25T04:00:13.146ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
792025-02-25T04:00:13.146ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
802025-02-25T04:00:13.146ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
812025-02-25T04:00:13.146ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
822025-02-25T04:00:13.146ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
832025-02-25T04:00:13.146ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
842025-02-25T04:00:13.146ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
852025-02-25T04:00:13.146ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
862025-02-25T04:00:13.149ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.871817ms iteration = 24 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
872025-02-25T04:00:13.149ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
882025-02-25T04:00:13.149ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
892025-02-25T04:00:13.150ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
902025-02-25T04:00:13.152ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
912025-02-25T04:00:13.152ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
922025-02-25T04:00:13.152ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
932025-02-25T04:00:13.152ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 7.458567ms iteration = 24 status = {"errors":[],"garbage_collect_requested":[]}
942025-02-25T04:00:13.152ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
952025-02-25T04:00:13.154ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
962025-02-25T04:00:13.154ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
972025-02-25T04:00:13.154ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
982025-02-25T04:00:13.160ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
992025-02-25T04:00:13.164ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
1002025-02-25T04:00:13.164ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
1012025-02-25T04:00:13.164ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
1022025-02-25T04:00:13.166ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 21.330247ms iteration = 24 status = {"errors":[],"finish_invoked_ok":[]}
1032025-02-25T04:00:13.167ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f193e77e-2472-41cb-b882-a3083cbe8c34.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f193e77e-2472-41cb-b882-a3083cbe8c34.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1671, tv_nsec: 549926883 } })
1042025-02-25T04:00:13.168ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1155
1052025-02-25T04:00:13.168ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f193e77e-2472-41cb-b882-a3083cbe8c34.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f193e77e-2472-41cb-b882-a3083cbe8c34.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1671, tv_nsec: 550690140 } })
1062025-02-25T04:00:13.169ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1155
1072025-02-25T04:00:13.170ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1082025-02-25T04:00:13.171ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
1092025-02-25T04:00:13.171ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
1102025-02-25T04:00:13.172ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
1112025-02-25T04:00:13.172ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
1122025-02-25T04:00:13.172ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
1132025-02-25T04:00:13.200ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
1142025-02-25T04:00:13.200ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
1152025-02-25T04:00:13.206ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1162025-02-25T04:00:13.208ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
1172025-02-25T04:00:13.208ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
1182025-02-25T04:00:13.208ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
1192025-02-25T04:00:13.208ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
1202025-02-25T04:00:13.210ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
1212025-02-25T04:00:13.210ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1222025-02-25T04:00:13.210ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1232025-02-25T04:00:13.210ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
1242025-02-25T04:00:13.210ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
1252025-02-25T04:00:13.210ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
1262025-02-25T04:00:13.211ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
1272025-02-25T04:00:13.211ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
1282025-02-25T04:00:13.211ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
1292025-02-25T04:00:13.211ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
1302025-02-25T04:00:13.212ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
1312025-02-25T04:00:13.212ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1322025-02-25T04:00:13.213ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1332025-02-25T04:00:13.214ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 69.048563ms iteration = 24 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
1342025-02-25T04:00:13.221ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 76.547251ms iteration = 24 status = {"errors":[],"requests_created_ok":[]}
1352025-02-25T04:00:13.223ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1362025-02-25T04:00:13.225ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
1372025-02-25T04:00:13.225ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
1382025-02-25T04:00:13.225ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
1392025-02-25T04:00:13.225ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1402025-02-25T04:00:13.226ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
1412025-02-25T04:00:13.226ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
1422025-02-25T04:00:13.226ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "9edc630e-183e-483b-84c7-6b0d1ddf0df6", "content-length": "26", "date": "Tue, 25 Feb 2025 04:00:13 GMT"} })
1432025-02-25T04:00:13.226ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
1442025-02-25T04:00:13.227ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
1452025-02-25T04:00:13.227ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
1462025-02-25T04:00:13.227ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
1472025-02-25T04:00:13.227ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
1482025-02-25T04:00:13.228ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
1492025-02-25T04:00:13.228ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "f511521c-bc7a-4441-bcd6-ccbaac43c22e", "content-length": "26", "date": "Tue, 25 Feb 2025 04:00:13 GMT"} })
1502025-02-25T04:00:13.228ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
1512025-02-25T04:00:13.229ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
1522025-02-25T04:00:13.229ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
1532025-02-25T04:00:13.229ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
1542025-02-25T04:00:13.229ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
1552025-02-25T04:00:13.229ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
1562025-02-25T04:00:13.229ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
1572025-02-25T04:00:13.231ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f193e77e-2472-41cb-b882-a3083cbe8c34.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f193e77e-2472-41cb-b882-a3083cbe8c34.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1671, tv_nsec: 613202308 } })
1582025-02-25T04:00:13.231ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1592025-02-25T04:00:13.231ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1155 rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8
1602025-02-25T04:00:13.232ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1612025-02-25T04:00:13.232ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1622025-02-25T04:00:13.234ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
1632025-02-25T04:00:13.234ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
1642025-02-25T04:00:13.234ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
1652025-02-25T04:00:13.235ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
1662025-02-25T04:00:13.235ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
1672025-02-25T04:00:13.235ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 84829298-e6df-4d09-ba9a-550ee8541f9e, lookup_type: ById(84829298-e6df-4d09-ba9a-550ee8541f9e) } result = Ok(())
1682025-02-25T04:00:13.235ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
1692025-02-25T04:00:13.235ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
1702025-02-25T04:00:13.235ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
1712025-02-25T04:00:13.238ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
1722025-02-25T04:00:13.238ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 93.532381ms iteration = 24 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
1732025-02-25T04:00:13.244ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 98.925772ms iteration = 70 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
1742025-02-25T04:00:13.245ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1752025-02-25T04:00:13.245ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1762025-02-25T04:00:13.248ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1772025-02-25T04:00:13.248ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1782025-02-25T04:00:13.255ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 84829298-e6df-4d09-ba9a-550ee8541f9e original_time_created = 2025-02-25 03:48:33.610641 UTC target_id = 84829298-e6df-4d09-ba9a-550ee8541f9e time_created = 2025-02-25 03:48:33.610641 UTC
1792025-02-25T04:00:13.255ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 110.305876ms iteration = 70 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"84829298-e6df-4d09-ba9a-550ee8541f9e","time_created":"2025-02-25T03:48:33.610641Z"}
1802025-02-25T04:00:13.259ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 113.886146ms iteration = 24 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
1812025-02-25T04:00:13.263ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
1822025-02-25T04:00:13.263ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
1832025-02-25T04:00:13.263ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 118.244374ms iteration = 24 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
1842025-02-25T04:00:13.272ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
1852025-02-25T04:00:13.272ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
1862025-02-25T04:00:13.274ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1872025-02-25T04:00:13.275ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
1882025-02-25T04:00:13.275ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
1892025-02-25T04:00:13.275ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 130.444256ms iteration = 24 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
1902025-02-25T04:00:13.275ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
1912025-02-25T04:00:13.295ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1922025-02-25T04:00:13.298ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: f193e77e-2472-41cb-b882-a3083cbe8c34, lookup_type: ById(f193e77e-2472-41cb-b882-a3083cbe8c34) } result = Ok(())
1932025-02-25T04:00:13.298ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1942025-02-25T04:00:13.298ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1952025-02-25T04:00:13.303ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
1962025-02-25T04:00:13.303ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
1972025-02-25T04:00:13.312ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 167.336562ms iteration = 24 status = {}
1982025-02-25T04:00:13.316ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 zone_address = fd00:1122:3344:101::2
1992025-02-25T04:00:13.316ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
2002025-02-25T04:00:13.316ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "9465dbe4-b088-42d0-9c6e-26c17c821d43", "content-length": "26", "date": "Tue, 25 Feb 2025 04:00:13 GMT"} })
2012025-02-25T04:00:13.316ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1168 rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
2022025-02-25T04:00:13.317ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:357
2032025-02-25T04:00:13.317ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 171.556766ms iteration = 24 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
2042025-02-25T04:00:13.318ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8
2052025-02-25T04:00:13.332ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
2062025-02-25T04:00:13.333ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "75f65853-a85b-4d57-9f57-27674cb9f66d", "content-length": "2", "date": "Tue, 25 Feb 2025 04:00:12 GMT"} })
2072025-02-25T04:00:13.334ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
2082025-02-25T04:00:13.334ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
2092025-02-25T04:00:13.363ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 217.76774ms iteration = 26 status = {"physical_disks_added":0}
2102025-02-25T04:00:13.386ZINFOnexus (BackgroundTasks): VPC route manager sled f193e77e-2472-41cb-b882-a3083cbe8c34 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
2112025-02-25T04:00:13.386ZDEBGnexus (BackgroundTasks): client request SledAgent = f193e77e-2472-41cb-b882-a3083cbe8c34 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
2122025-02-25T04:00:13.388ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 242.890262ms iteration = 24 status = 0
2132025-02-25T04:00:13.388ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 243.091246ms iteration = 24 status = null
2142025-02-25T04:00:13.423ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
2152025-02-25T04:00:13.424ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
2162025-02-25T04:00:13.425ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2172025-02-25T04:00:13.427ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
2182025-02-25T04:00:13.427ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
2192025-02-25T04:00:13.427ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
2202025-02-25T04:00:13.429ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2212025-02-25T04:00:13.431ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
2222025-02-25T04:00:13.431ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
2232025-02-25T04:00:13.431ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
2242025-02-25T04:00:13.477ZDEBGnexus (BackgroundTasks): client response SledAgent = f193e77e-2472-41cb-b882-a3083cbe8c34 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "55e2cdbc-c487-476c-9f6c-3333f2c278f9", "content-length": "372", "date": "Tue, 25 Feb 2025 04:00:13 GMT"} })
2252025-02-25T04:00:13.477ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
2262025-02-25T04:00:13.477ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
2272025-02-25T04:00:13.492ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
2282025-02-25T04:00:13.492ZDEBGnexus (BackgroundTasks): client request SledAgent = f193e77e-2472-41cb-b882-a3083cbe8c34 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
2292025-02-25T04:00:13.492ZDEBGnexus (BackgroundTasks): client request SledAgent = f193e77e-2472-41cb-b882-a3083cbe8c34 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
2302025-02-25T04:00:13.493ZDEBGnexus (BackgroundTasks): client response SledAgent = f193e77e-2472-41cb-b882-a3083cbe8c34 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "43a3e211-8ee1-40ed-9c57-195c7b94930c", "date": "Tue, 25 Feb 2025 04:00:13 GMT"} })
2312025-02-25T04:00:13.493ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
2322025-02-25T04:00:13.493ZDEBGnexus (BackgroundTasks): client response SledAgent = f193e77e-2472-41cb-b882-a3083cbe8c34 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "a69047a0-ee62-4163-b3e6-931343bd2c75", "content-length": "691", "date": "Tue, 25 Feb 2025 04:00:13 GMT"} })
2332025-02-25T04:00:13.493ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
2342025-02-25T04:00:13.493ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
2352025-02-25T04:00:13.493ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
2362025-02-25T04:00:13.493ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
2372025-02-25T04:00:13.494ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 349.156658ms iteration = 24 status = {}
2382025-02-25T04:00:13.495ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
2392025-02-25T04:00:13.495ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
2402025-02-25T04:00:13.495ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
2412025-02-25T04:00:13.496ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
2422025-02-25T04:00:13.497ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: aa10810a-fc5e-4e32-a4d1-f8dbfe9cd7f7, rack_id: 511fa71f-9d49-431d-90d0-c34f01fc4ef8, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(7594d5a9-a289-4d08-9933-529ac715d7e5) } rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 switch_location = Switch0
2432025-02-25T04:00:13.497ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
2442025-02-25T04:00:13.497ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
2452025-02-25T04:00:13.499ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
2462025-02-25T04:00:13.499ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
2472025-02-25T04:00:13.502ZINFOnexus (BackgroundTasks): VPC route manager sled f193e77e-2472-41cb-b882-a3083cbe8c34 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
2482025-02-25T04:00:13.502ZDEBGnexus (BackgroundTasks): client request SledAgent = f193e77e-2472-41cb-b882-a3083cbe8c34 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
2492025-02-25T04:00:13.504ZDEBGnexus (BackgroundTasks): client response SledAgent = f193e77e-2472-41cb-b882-a3083cbe8c34 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "8d7f4cab-55df-4d42-aaa9-da6e0e00261c", "date": "Tue, 25 Feb 2025 04:00:13 GMT"} })
2502025-02-25T04:00:13.506ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 360.956332ms iteration = 24 status = {}
2512025-02-25T04:00:13.507ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
2522025-02-25T04:00:13.507ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "15ff0997-d6ea-4709-97ce-d1915bbe84a5", "content-length": "134", "date": "Tue, 25 Feb 2025 04:00:12 GMT"} })
2532025-02-25T04:00:13.507ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
2542025-02-25T04:00:13.507ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 7594d5a9-a289-4d08-9933-529ac715d7e5
2552025-02-25T04:00:13.507ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
2562025-02-25T04:00:13.507ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
2572025-02-25T04:00:13.507ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
2582025-02-25T04:00:13.507ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
2592025-02-25T04:00:13.507ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
2602025-02-25T04:00:13.507ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
2612025-02-25T04:00:13.507ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
2622025-02-25T04:00:13.507ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
2632025-02-25T04:00:13.508ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8
2642025-02-25T04:00:13.508ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
2652025-02-25T04:00:13.508ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
2662025-02-25T04:00:13.510ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
2672025-02-25T04:00:13.510ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c2f607b-ca89-45ab-ab17-908b39bf1dfb", "content-length": "2", "date": "Tue, 25 Feb 2025 04:00:12 GMT"} })
2682025-02-25T04:00:13.510ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
2692025-02-25T04:00:13.510ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "26234dde-a4bc-43ca-a237-5b90b2322064", "content-length": "52", "date": "Tue, 25 Feb 2025 04:00:12 GMT"} })
2702025-02-25T04:00:13.510ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 switch_location = Switch0
2712025-02-25T04:00:13.510ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
2722025-02-25T04:00:13.662ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "b6dd4bb6-7513-4c65-856a-e4ff180606b9", "date": "Tue, 25 Feb 2025 04:00:13 GMT"} })
2732025-02-25T04:00:13.662ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 switch = Switch0
2742025-02-25T04:00:13.662ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
2752025-02-25T04:00:13.662ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "439c0f3d-3644-4191-ab52-7e1df62f7b30", "content-length": "682", "date": "Tue, 25 Feb 2025 04:00:13 GMT"} })
2762025-02-25T04:00:13.662ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
2772025-02-25T04:00:13.663ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
2782025-02-25T04:00:13.666ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
2792025-02-25T04:00:13.667ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
2802025-02-25T04:00:13.670ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
2812025-02-25T04:00:13.670ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
2822025-02-25T04:00:13.672ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
2832025-02-25T04:00:13.672ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
2842025-02-25T04:00:13.679ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 534.075903ms iteration = 24 status = {}
2852025-02-25T04:00:15.635ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.15.1/src/server.rs:867 latency_us = 367 local_addr = [fd00:1122:3344:101::c]:52384 method = GET remote_addr = [fd00:1122:3344:101::d]:48316 req_id = ed5d1153-b8f9-48d8-a136-1d3301bcf89c response_code = 200 uri = /d8b217e1-a0a4-419c-82c9-30f4d54f8fbd
2862025-02-25T04:00:23.144ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 71 reason = Timeout
2872025-02-25T04:00:23.144ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2882025-02-25T04:00:23.144ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 71 reason = Timeout
2892025-02-25T04:00:23.144ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
2902025-02-25T04:00:23.145ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2912025-02-25T04:00:23.145ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
2922025-02-25T04:00:23.146ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2932025-02-25T04:00:23.147ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
2942025-02-25T04:00:23.147ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
2952025-02-25T04:00:23.149ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.635523ms iteration = 71 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
2962025-02-25T04:00:23.149ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
2972025-02-25T04:00:23.149ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2982025-02-25T04:00:23.149ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2992025-02-25T04:00:23.151ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3002025-02-25T04:00:23.151ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3012025-02-25T04:00:23.153ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3022025-02-25T04:00:23.155ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 84829298-e6df-4d09-ba9a-550ee8541f9e, lookup_type: ById(84829298-e6df-4d09-ba9a-550ee8541f9e) } result = Ok(())
3032025-02-25T04:00:23.155ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3042025-02-25T04:00:23.155ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3052025-02-25T04:00:23.166ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 84829298-e6df-4d09-ba9a-550ee8541f9e original_time_created = 2025-02-25 03:48:33.610641 UTC target_id = 84829298-e6df-4d09-ba9a-550ee8541f9e time_created = 2025-02-25 03:48:33.610641 UTC
3062025-02-25T04:00:23.166ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.184426ms iteration = 71 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"84829298-e6df-4d09-ba9a-550ee8541f9e","time_created":"2025-02-25T03:48:33.610641Z"}
3072025-02-25T04:00:25.635ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.15.1/src/server.rs:867 latency_us = 399 local_addr = [fd00:1122:3344:101::c]:52384 method = GET remote_addr = [fd00:1122:3344:101::d]:48316 req_id = 421e391e-d1a4-46af-aa2e-1698cc1e5059 response_code = 200 uri = /d8b217e1-a0a4-419c-82c9-30f4d54f8fbd
3082025-02-25T04:00:33.144ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 72 reason = Timeout
3092025-02-25T04:00:33.144ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 72 reason = Timeout
3102025-02-25T04:00:33.144ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
3112025-02-25T04:00:33.144ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3122025-02-25T04:00:33.145ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
3132025-02-25T04:00:33.145ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3142025-02-25T04:00:33.146ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
3152025-02-25T04:00:33.147ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
3162025-02-25T04:00:33.147ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3172025-02-25T04:00:33.150ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
3182025-02-25T04:00:33.150ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3192025-02-25T04:00:33.150ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3202025-02-25T04:00:33.152ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3212025-02-25T04:00:33.152ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3222025-02-25T04:00:33.154ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3232025-02-25T04:00:33.156ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 84829298-e6df-4d09-ba9a-550ee8541f9e, lookup_type: ById(84829298-e6df-4d09-ba9a-550ee8541f9e) } result = Ok(())
3242025-02-25T04:00:33.156ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3252025-02-25T04:00:33.156ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3262025-02-25T04:00:33.168ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 84829298-e6df-4d09-ba9a-550ee8541f9e original_time_created = 2025-02-25 03:48:33.610641 UTC target_id = 84829298-e6df-4d09-ba9a-550ee8541f9e time_created = 2025-02-25 03:48:33.610641 UTC
3272025-02-25T04:00:33.168ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 23.422984ms iteration = 72 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"84829298-e6df-4d09-ba9a-550ee8541f9e","time_created":"2025-02-25T03:48:33.610641Z"}
3282025-02-25T04:00:33.177ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 32.848915ms iteration = 72 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
3292025-02-25T04:00:35.635ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.15.1/src/server.rs:867 latency_us = 322 local_addr = [fd00:1122:3344:101::c]:52384 method = GET remote_addr = [fd00:1122:3344:101::d]:48316 req_id = d2f05858-22c1-4d2f-9c59-fff43c77fb5c response_code = 200 uri = /d8b217e1-a0a4-419c-82c9-30f4d54f8fbd
3302025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 73 reason = Timeout
3312025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
3322025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 6 reason = Timeout
3332025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 13 reason = Timeout
3342025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
3352025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
3362025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
3372025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 13 reason = Timeout
3382025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 25 reason = Timeout
3392025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
3402025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 25 reason = Timeout
3412025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
3422025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 25 reason = Timeout
3432025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
3442025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
3452025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 13 reason = Timeout
3462025-02-25T04:00:43.145ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-02-25 03:50:43.145392423 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
3472025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
3482025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 14 reason = Timeout
3492025-02-25T04:00:43.145ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = 84829298-e6df-4d09-ba9a-550ee8541f9e
3502025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 14.49\u{b5}s iteration = 14 status = {"enabled":false,"target_id":"84829298-e6df-4d09-ba9a-550ee8541f9e"}
3512025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 13 reason = Timeout
3522025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 15 reason = Timeout
3532025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
3542025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 25 reason = Timeout
3552025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 25 reason = Timeout
3562025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
3572025-02-25T04:00:43.145ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
3582025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 73 reason = Timeout
3592025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
3602025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3612025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 13 reason = Timeout
3622025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
3632025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 13 reason = Timeout
3642025-02-25T04:00:43.145ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
3652025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
3662025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 13 reason = Timeout
3672025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 25 reason = Timeout
3682025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
3692025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 25 reason = Timeout
3702025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 25 reason = Timeout
3712025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
3722025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 27 reason = Timeout
3732025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 25 reason = Timeout
3742025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 15 reason = Timeout
3752025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
3762025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 25 reason = Timeout
3772025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
3782025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
3792025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 25 reason = Timeout
3802025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
3812025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 25 reason = Timeout
3822025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
3832025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 13 reason = Timeout
3842025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
3852025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 25 reason = Timeout
3862025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
3872025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
3882025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 13 reason = Timeout
3892025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
3902025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
3912025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 25 reason = Timeout
3922025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
3932025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 25 reason = Timeout
3942025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
3952025-02-25T04:00:43.145ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 25 reason = Timeout
3962025-02-25T04:00:43.146ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 13 reason = Timeout
3972025-02-25T04:00:43.146ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
3982025-02-25T04:00:43.146ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
3992025-02-25T04:00:43.146ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
4002025-02-25T04:00:43.146ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
4012025-02-25T04:00:43.146ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
4022025-02-25T04:00:43.146ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
4032025-02-25T04:00:43.146ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
4042025-02-25T04:00:43.146ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
4052025-02-25T04:00:43.146ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4062025-02-25T04:00:43.146ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
4072025-02-25T04:00:43.146ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
4082025-02-25T04:00:43.146ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
4092025-02-25T04:00:43.146ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
4102025-02-25T04:00:43.146ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
4112025-02-25T04:00:43.146ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
4122025-02-25T04:00:43.146ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
4132025-02-25T04:00:43.146ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
4142025-02-25T04:00:43.146ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
4152025-02-25T04:00:43.146ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
4162025-02-25T04:00:43.146ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
4172025-02-25T04:00:43.146ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
4182025-02-25T04:00:43.146ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
4192025-02-25T04:00:43.146ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
4202025-02-25T04:00:43.146ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
4212025-02-25T04:00:43.146ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
4222025-02-25T04:00:43.146ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
4232025-02-25T04:00:43.146ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
4242025-02-25T04:00:43.146ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
4252025-02-25T04:00:43.146ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
4262025-02-25T04:00:43.146ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
4272025-02-25T04:00:43.146ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("01b5d6b1-1b71-43e4-a271-a3a6806529b6.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("1317d5c6-aaa1-43a7-acc5-dddf70c33e69.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("1317d5c6-aaa1-43a7-acc5-dddf70c33e69.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 1701, tv_nsec: 528476894 } })
4282025-02-25T04:00:43.146ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("45e0a6d8-360c-43f4-92e2-67ec17cbbadc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("53b0aac1-c323-4328-a42b-f561d7626f18.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("b8656c30-4b64-41c6-88c8-f869cc289571.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("b8656c30-4b64-41c6-88c8-f869cc289571.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 1701, tv_nsec: 528489134 } })
4292025-02-25T04:00:43.146ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
4302025-02-25T04:00:43.146ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
4312025-02-25T04:00:43.146ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
4322025-02-25T04:00:43.146ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
4332025-02-25T04:00:43.146ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
4342025-02-25T04:00:43.146ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
4352025-02-25T04:00:43.146ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.320159ms iteration = 13 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
4362025-02-25T04:00:43.146ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
4372025-02-25T04:00:43.147ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
4382025-02-25T04:00:43.147ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.681057ms iteration = 13 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353"]}
4392025-02-25T04:00:43.147ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
4402025-02-25T04:00:43.147ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
4412025-02-25T04:00:43.148ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
4422025-02-25T04:00:43.148ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 2.67223ms iteration = 25 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
4432025-02-25T04:00:43.148ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
4442025-02-25T04:00:43.149ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4452025-02-25T04:00:43.150ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-02-25 03:50:43.145392423 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
4462025-02-25T04:00:43.150ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 5.149006ms iteration = 13 status = {"expiration":"2025-02-25T03:50:43.145392423Z","pruned":[]}
4472025-02-25T04:00:43.151ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4482025-02-25T04:00:43.151ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4492025-02-25T04:00:43.151ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f193e77e-2472-41cb-b882-a3083cbe8c34.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f193e77e-2472-41cb-b882-a3083cbe8c34.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1701, tv_nsec: 533258990 } })
4502025-02-25T04:00:43.151ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1155
4512025-02-25T04:00:43.151ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
4522025-02-25T04:00:43.151ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
4532025-02-25T04:00:43.151ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
4542025-02-25T04:00:43.153ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
4552025-02-25T04:00:43.153ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
4562025-02-25T04:00:43.154ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 8.587192ms iteration = 13 status = {"errors":[],"found_port_ok":[]}
4572025-02-25T04:00:43.154ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
4582025-02-25T04:00:43.154ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
4592025-02-25T04:00:43.154ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4602025-02-25T04:00:43.154ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4612025-02-25T04:00:43.155ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4622025-02-25T04:00:43.158ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4632025-02-25T04:00:43.158ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
4642025-02-25T04:00:43.158ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
4652025-02-25T04:00:43.159ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
4662025-02-25T04:00:43.160ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
4672025-02-25T04:00:43.160ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
4682025-02-25T04:00:43.160ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
4692025-02-25T04:00:43.160ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
4702025-02-25T04:00:43.160ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
4712025-02-25T04:00:43.161ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
4722025-02-25T04:00:43.161ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
4732025-02-25T04:00:43.161ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f193e77e-2472-41cb-b882-a3083cbe8c34.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f193e77e-2472-41cb-b882-a3083cbe8c34.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1701, tv_nsec: 543589032 } })
4742025-02-25T04:00:43.161ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1155
4752025-02-25T04:00:43.163ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4762025-02-25T04:00:43.164ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4772025-02-25T04:00:43.166ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
4782025-02-25T04:00:43.166ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
4792025-02-25T04:00:43.166ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4802025-02-25T04:00:43.166ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
4812025-02-25T04:00:43.167ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
4822025-02-25T04:00:43.167ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
4832025-02-25T04:00:43.168ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4842025-02-25T04:00:43.168ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
4852025-02-25T04:00:43.168ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
4862025-02-25T04:00:43.169ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
4872025-02-25T04:00:43.169ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
4882025-02-25T04:00:43.169ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
4892025-02-25T04:00:43.169ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
4902025-02-25T04:00:43.169ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
4912025-02-25T04:00:43.169ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
4922025-02-25T04:00:43.170ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
4932025-02-25T04:00:43.170ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
4942025-02-25T04:00:43.170ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
4952025-02-25T04:00:43.171ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 26.122844ms iteration = 25 status = {"errors":[],"finish_invoked_ok":[]}
4962025-02-25T04:00:43.172ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4972025-02-25T04:00:43.173ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4982025-02-25T04:00:43.174ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4992025-02-25T04:00:43.175ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
5002025-02-25T04:00:43.175ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
5012025-02-25T04:00:43.175ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
5022025-02-25T04:00:43.176ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5032025-02-25T04:00:43.176ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
5042025-02-25T04:00:43.176ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
5052025-02-25T04:00:43.176ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
5062025-02-25T04:00:43.177ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 31.340781ms iteration = 25 status = {"errors":[],"requests_created_ok":[]}
5072025-02-25T04:00:43.179ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
5082025-02-25T04:00:43.179ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
5092025-02-25T04:00:43.179ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
5102025-02-25T04:00:43.179ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5112025-02-25T04:00:43.180ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
5122025-02-25T04:00:43.182ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
5132025-02-25T04:00:43.182ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
5142025-02-25T04:00:43.182ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
5152025-02-25T04:00:43.188ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5162025-02-25T04:00:43.189ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5172025-02-25T04:00:43.190ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
5182025-02-25T04:00:43.190ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
5192025-02-25T04:00:43.190ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
5202025-02-25T04:00:43.194ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
5212025-02-25T04:00:43.194ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
5222025-02-25T04:00:43.194ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
5232025-02-25T04:00:43.195ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
5242025-02-25T04:00:43.195ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
5252025-02-25T04:00:43.195ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
5262025-02-25T04:00:43.196ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5272025-02-25T04:00:43.196ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
5282025-02-25T04:00:43.196ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5292025-02-25T04:00:43.199ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
5302025-02-25T04:00:43.199ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
5312025-02-25T04:00:43.199ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
5322025-02-25T04:00:43.200ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
5332025-02-25T04:00:43.201ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-f193e77e-2472-41cb-b882-a3083cbe8c34.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-f193e77e-2472-41cb-b882-a3083cbe8c34.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1701, tv_nsec: 583348511 } })
5342025-02-25T04:00:43.201ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1155 rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8
5352025-02-25T04:00:43.204ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5362025-02-25T04:00:43.208ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 84829298-e6df-4d09-ba9a-550ee8541f9e, lookup_type: ById(84829298-e6df-4d09-ba9a-550ee8541f9e) } result = Ok(())
5372025-02-25T04:00:43.208ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
5382025-02-25T04:00:43.209ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
5392025-02-25T04:00:43.211ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5402025-02-25T04:00:43.214ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 68.543402ms iteration = 25 status = {"errors":[],"garbage_collect_requested":[]}
5412025-02-25T04:00:43.215ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
5422025-02-25T04:00:43.215ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
5432025-02-25T04:00:43.216ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
5442025-02-25T04:00:43.217ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5452025-02-25T04:00:43.219ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
5462025-02-25T04:00:43.220ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
5472025-02-25T04:00:43.220ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
5482025-02-25T04:00:43.220ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
5492025-02-25T04:00:43.220ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
5502025-02-25T04:00:43.222ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
5512025-02-25T04:00:43.223ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
5522025-02-25T04:00:43.223ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
5532025-02-25T04:00:43.223ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
5542025-02-25T04:00:43.229ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5552025-02-25T04:00:43.231ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 84829298-e6df-4d09-ba9a-550ee8541f9e original_time_created = 2025-02-25 03:48:33.610641 UTC target_id = 84829298-e6df-4d09-ba9a-550ee8541f9e time_created = 2025-02-25 03:48:33.610641 UTC
5562025-02-25T04:00:43.231ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 85.688685ms iteration = 73 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"84829298-e6df-4d09-ba9a-550ee8541f9e","time_created":"2025-02-25T03:48:33.610641Z"}
5572025-02-25T04:00:43.232ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
5582025-02-25T04:00:43.232ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
5592025-02-25T04:00:43.232ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
5602025-02-25T04:00:43.232ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
5612025-02-25T04:00:43.232ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:171 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
5622025-02-25T04:00:43.232ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 86.789311ms iteration = 13 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
5632025-02-25T04:00:43.233ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 87.321202ms iteration = 25 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
5642025-02-25T04:00:43.233ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 88.190491ms iteration = 73 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
5652025-02-25T04:00:43.234ZDEBGnexus (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 89.01394ms iteration = 6 status = {"nsuccess":5}
5662025-02-25T04:00:43.237ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5672025-02-25T04:00:43.237ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5682025-02-25T04:00:43.238ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
5692025-02-25T04:00:43.238ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
5702025-02-25T04:00:43.238ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
5712025-02-25T04:00:43.238ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 92.802525ms iteration = 13 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
5722025-02-25T04:00:43.238ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
5732025-02-25T04:00:43.239ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 93.658474ms iteration = 25 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
5742025-02-25T04:00:43.240ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5752025-02-25T04:00:43.243ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
5762025-02-25T04:00:43.243ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
5772025-02-25T04:00:43.243ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
5782025-02-25T04:00:43.243ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5792025-02-25T04:00:43.243ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5802025-02-25T04:00:43.246ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
5812025-02-25T04:00:43.246ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
5822025-02-25T04:00:43.246ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
5832025-02-25T04:00:43.246ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
5842025-02-25T04:00:43.246ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
5852025-02-25T04:00:43.248ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5862025-02-25T04:00:43.249ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
5872025-02-25T04:00:43.249ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
5882025-02-25T04:00:43.249ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
5892025-02-25T04:00:43.249ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
5902025-02-25T04:00:43.249ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
5912025-02-25T04:00:43.251ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5922025-02-25T04:00:43.251ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
5932025-02-25T04:00:43.251ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
5942025-02-25T04:00:43.251ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 105.870907ms iteration = 25 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
5952025-02-25T04:00:43.251ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5962025-02-25T04:00:43.252ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
5972025-02-25T04:00:43.252ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
5982025-02-25T04:00:43.253ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: f193e77e-2472-41cb-b882-a3083cbe8c34, lookup_type: ById(f193e77e-2472-41cb-b882-a3083cbe8c34) } result = Ok(())
5992025-02-25T04:00:43.253ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
6002025-02-25T04:00:43.254ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
6012025-02-25T04:00:43.254ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
6022025-02-25T04:00:43.254ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
6032025-02-25T04:00:43.254ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
6042025-02-25T04:00:43.254ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
6052025-02-25T04:00:43.254ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 109.448798ms iteration = 25 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
6062025-02-25T04:00:43.256ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
6072025-02-25T04:00:43.256ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
6082025-02-25T04:00:43.256ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
6092025-02-25T04:00:43.257ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 111.648567ms iteration = 25 status = 0
6102025-02-25T04:00:43.257ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
6112025-02-25T04:00:43.257ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 8eba5ccb-30ef-4afa-ac24-8cae617b9880 dns_zone_name = oxide.test found_so_far = 0 version = 2
6122025-02-25T04:00:43.257ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
6132025-02-25T04:00:43.258ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
6142025-02-25T04:00:43.263ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6152025-02-25T04:00:43.264ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
6162025-02-25T04:00:43.265ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
6172025-02-25T04:00:43.267ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6182025-02-25T04:00:43.267ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
6192025-02-25T04:00:43.267ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
6202025-02-25T04:00:43.267ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
6212025-02-25T04:00:43.269ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 8eba5ccb-30ef-4afa-ac24-8cae617b9880 dns_zone_name = oxide.test found_so_far = 1 version = 2
6222025-02-25T04:00:43.270ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
6232025-02-25T04:00:43.270ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-02-25 03:49:12.391510 UTC dns_group = external generation = 2
6242025-02-25T04:00:43.270ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 124.709419ms iteration = 13 status = {"generation":2}
6252025-02-25T04:00:43.270ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
6262025-02-25T04:00:43.270ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
6272025-02-25T04:00:43.270ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
6282025-02-25T04:00:43.270ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 124.871573ms iteration = 25 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
6292025-02-25T04:00:43.275ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
6302025-02-25T04:00:43.284ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 139.201683ms iteration = 27 status = {"physical_disks_added":0}
6312025-02-25T04:00:43.287ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:357
6322025-02-25T04:00:43.287ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 141.831302ms iteration = 25 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
6332025-02-25T04:00:43.300ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
6342025-02-25T04:00:43.300ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = bed1abb3-5448-46e6-a663-ee38675a84d1 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
6352025-02-25T04:00:43.300ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
6362025-02-25T04:00:43.300ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
6372025-02-25T04:00:43.304ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
6382025-02-25T04:00:43.304ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
6392025-02-25T04:00:43.304ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "64f0f6cd-5c6d-4053-b182-17528f707d10", "content-length": "26", "date": "Tue, 25 Feb 2025 04:00:43 GMT"} })
6402025-02-25T04:00:43.304ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
6412025-02-25T04:00:43.307ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
6422025-02-25T04:00:43.313ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
6432025-02-25T04:00:43.315ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
6442025-02-25T04:00:43.315ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
6452025-02-25T04:00:43.321ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-02-25T03:48:40.896126Z, time_modified: 2025-02-25T03:48:40.896126Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 7a265c65-87fe-43ef-92b1-187ae9275136, db_silo: Silo { identity: SiloIdentity { id: 7a265c65-87fe-43ef-92b1-187ae9275136, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-02-25T03:49:12.314272Z, time_modified: 2025-02-25T03:49:12.314272Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "a0a10d9d562cda3d37dbd411c72c853739450b2d6959511fd447e68280ddc765" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 7a265c65-87fe-43ef-92b1-187ae9275136, db_silo: Silo { identity: SiloIdentity { id: 7a265c65-87fe-43ef-92b1-187ae9275136, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-02-25T03:49:12.314272Z, time_modified: 2025-02-25T03:49:12.314272Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "a0a10d9d562cda3d37dbd411c72c853739450b2d6959511fd447e68280ddc765" }] }) }
6462025-02-25T04:00:43.321ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 175.975225ms iteration = 13 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"7a265c65-87fe-43ef-92b1-187ae9275136","tls_certs":["a0a10d9d562cda3d37dbd411c72c853739450b2d6959511fd447e68280ddc765"]}},"default_endpoint":{"silo_id":"7a265c65-87fe-43ef-92b1-187ae9275136","tls_certs":["a0a10d9d562cda3d37dbd411c72c853739450b2d6959511fd447e68280ddc765"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
6472025-02-25T04:00:43.334ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6482025-02-25T04:00:43.338ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
6492025-02-25T04:00:43.338ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
6502025-02-25T04:00:43.338ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
6512025-02-25T04:00:43.341ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
6522025-02-25T04:00:43.341ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 196.227078ms iteration = 13 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
6532025-02-25T04:00:43.410ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
6542025-02-25T04:00:43.410ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "bf7a022b-533a-465e-a720-46cdfde5257d", "date": "Tue, 25 Feb 2025 04:00:43 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
6552025-02-25T04:00:43.410ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
6562025-02-25T04:00:43.413ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "fde85b09-3683-4787-b600-fe9b44a5bc4c", "date": "Tue, 25 Feb 2025 04:00:43 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
6572025-02-25T04:00:43.413ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
6582025-02-25T04:00:43.414ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
6592025-02-25T04:00:43.414ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 269.21781ms iteration = 15 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
6602025-02-25T04:00:43.437ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1163 zone_address = fd00:1122:3344:101::2
6612025-02-25T04:00:43.437ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
6622025-02-25T04:00:43.437ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d64ff4c-4566-4c69-83a2-d93142be534b", "content-length": "26", "date": "Tue, 25 Feb 2025 04:00:43 GMT"} })
6632025-02-25T04:00:43.437ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1168 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
6642025-02-25T04:00:43.439ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
6652025-02-25T04:00:43.451ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1163 rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 zone_address = fd00:1122:3344:101::2
6662025-02-25T04:00:43.451ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
6672025-02-25T04:00:43.451ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "25e32f24-8f34-4944-bc27-b7a62428945c", "content-length": "26", "date": "Tue, 25 Feb 2025 04:00:43 GMT"} })
6682025-02-25T04:00:43.451ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1168 rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
6692025-02-25T04:00:43.453ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1200 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8
6702025-02-25T04:00:43.481ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 335.866711ms iteration = 25 status = {}
6712025-02-25T04:00:43.481ZINFOnexus (BackgroundTasks): VPC route manager sled f193e77e-2472-41cb-b882-a3083cbe8c34 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
6722025-02-25T04:00:43.481ZDEBGnexus (BackgroundTasks): client request SledAgent = f193e77e-2472-41cb-b882-a3083cbe8c34 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
6732025-02-25T04:00:43.482ZDEBGnexus (BackgroundTasks): client response SledAgent = f193e77e-2472-41cb-b882-a3083cbe8c34 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "896ac330-4882-4d43-9066-13c3c5404246", "content-length": "372", "date": "Tue, 25 Feb 2025 04:00:42 GMT"} })
6742025-02-25T04:00:43.482ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
6752025-02-25T04:00:43.482ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
6762025-02-25T04:00:43.482ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
6772025-02-25T04:00:43.485ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = bed1abb3-5448-46e6-a663-ee38675a84d1 dns_zone_name = control-plane.oxide.internal found_so_far = 51 version = 1
6782025-02-25T04:00:43.485ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
6792025-02-25T04:00:43.485ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-02-25 03:49:11.889892 UTC dns_group = internal generation = 1
6802025-02-25T04:00:43.485ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 339.41721ms iteration = 13 status = {"generation":1}
6812025-02-25T04:00:43.532ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
6822025-02-25T04:00:43.532ZDEBGnexus (BackgroundTasks): client request SledAgent = f193e77e-2472-41cb-b882-a3083cbe8c34 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
6832025-02-25T04:00:43.532ZDEBGnexus (BackgroundTasks): client response SledAgent = f193e77e-2472-41cb-b882-a3083cbe8c34 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "1d1814d3-13e9-41e7-8c96-6cd5b863376e", "date": "Tue, 25 Feb 2025 04:00:43 GMT"} })
6842025-02-25T04:00:43.532ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
6852025-02-25T04:00:43.532ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
6862025-02-25T04:00:43.535ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
6872025-02-25T04:00:43.535ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
6882025-02-25T04:00:43.536ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
6892025-02-25T04:00:43.536ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "f719132f-4828-420e-b7ee-14a756bc3ca0", "content-length": "2", "date": "Tue, 25 Feb 2025 04:00:43 GMT"} })
6902025-02-25T04:00:43.537ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
6912025-02-25T04:00:43.538ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
6922025-02-25T04:00:43.538ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
6932025-02-25T04:00:43.538ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
6942025-02-25T04:00:43.540ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 394.451361ms iteration = 25 status = null
6952025-02-25T04:00:43.540ZINFOnexus (BackgroundTasks): VPC route manager sled f193e77e-2472-41cb-b882-a3083cbe8c34 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
6962025-02-25T04:00:43.540ZDEBGnexus (BackgroundTasks): client request SledAgent = f193e77e-2472-41cb-b882-a3083cbe8c34 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
6972025-02-25T04:00:43.562ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
6982025-02-25T04:00:43.563ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "68059dd9-f636-4529-aeb6-954bf96a746c", "date": "Tue, 25 Feb 2025 04:00:43 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
6992025-02-25T04:00:43.563ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
7002025-02-25T04:00:43.564ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "473c9514-5f8c-45a1-ad73-ebb572c4195c", "date": "Tue, 25 Feb 2025 04:00:43 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
7012025-02-25T04:00:43.564ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
7022025-02-25T04:00:43.565ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "cdf562f7-3336-467d-97d3-5878caeaffb1", "date": "Tue, 25 Feb 2025 04:00:43 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
7032025-02-25T04:00:43.565ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
7042025-02-25T04:00:43.567ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
7052025-02-25T04:00:43.567ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 421.850905ms iteration = 15 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
7062025-02-25T04:00:43.589ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
7072025-02-25T04:00:43.589ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
7082025-02-25T04:00:43.591ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7092025-02-25T04:00:43.592ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
7102025-02-25T04:00:43.592ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
7112025-02-25T04:00:43.592ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
7122025-02-25T04:00:43.619ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
7132025-02-25T04:00:43.620ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
7142025-02-25T04:00:43.622ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: aa10810a-fc5e-4e32-a4d1-f8dbfe9cd7f7, rack_id: 511fa71f-9d49-431d-90d0-c34f01fc4ef8, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(7594d5a9-a289-4d08-9933-529ac715d7e5) } rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 switch_location = Switch0
7152025-02-25T04:00:43.622ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
7162025-02-25T04:00:43.622ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
7172025-02-25T04:00:43.624ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7182025-02-25T04:00:43.625ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
7192025-02-25T04:00:43.625ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
7202025-02-25T04:00:43.625ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
7212025-02-25T04:00:43.629ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
7222025-02-25T04:00:43.629ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "db7bd11e-8e6d-40ed-803b-1053f996f53f", "content-length": "134", "date": "Tue, 25 Feb 2025 04:00:43 GMT"} })
7232025-02-25T04:00:43.629ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
7242025-02-25T04:00:43.629ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 7594d5a9-a289-4d08-9933-529ac715d7e5
7252025-02-25T04:00:43.630ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
7262025-02-25T04:00:43.630ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
7272025-02-25T04:00:43.630ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
7282025-02-25T04:00:43.630ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
7292025-02-25T04:00:43.630ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
7302025-02-25T04:00:43.630ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
7312025-02-25T04:00:43.630ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
7322025-02-25T04:00:43.630ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
7332025-02-25T04:00:43.630ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8
7342025-02-25T04:00:43.630ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
7352025-02-25T04:00:43.630ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
7362025-02-25T04:00:43.632ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
7372025-02-25T04:00:43.632ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "040a13d5-a004-4094-b369-3e1d6bbe0043", "content-length": "2", "date": "Tue, 25 Feb 2025 04:00:43 GMT"} })
7382025-02-25T04:00:43.632ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
7392025-02-25T04:00:43.632ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "866a5858-fde1-4676-8cb4-8a3fc0c8ade5", "content-length": "52", "date": "Tue, 25 Feb 2025 04:00:43 GMT"} })
7402025-02-25T04:00:43.632ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 switch_location = Switch0
7412025-02-25T04:00:43.632ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
7422025-02-25T04:00:43.652ZDEBGnexus (BackgroundTasks): client response SledAgent = f193e77e-2472-41cb-b882-a3083cbe8c34 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "37637fd8-045f-4b10-8841-00d1b821648c", "date": "Tue, 25 Feb 2025 04:00:43 GMT"} })
7432025-02-25T04:00:43.654ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 509.429423ms iteration = 25 status = {}
7442025-02-25T04:00:43.659ZDEBGnexus (BackgroundTasks): client request SledAgent = f193e77e-2472-41cb-b882-a3083cbe8c34 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
7452025-02-25T04:00:43.660ZDEBGnexus (BackgroundTasks): client response SledAgent = f193e77e-2472-41cb-b882-a3083cbe8c34 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "13b2b6ef-4889-4737-bdcd-b988fccce533", "content-length": "691", "date": "Tue, 25 Feb 2025 04:00:43 GMT"} })
7462025-02-25T04:00:43.660ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
7472025-02-25T04:00:43.660ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
7482025-02-25T04:00:43.660ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
7492025-02-25T04:00:43.662ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 516.90037ms iteration = 25 status = {}
7502025-02-25T04:00:43.839ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "933386ce-da5a-4ea0-8ef3-b1af587536b9", "date": "Tue, 25 Feb 2025 04:00:43 GMT"} })
7512025-02-25T04:00:43.839ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 switch = Switch0
7522025-02-25T04:00:43.839ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
7532025-02-25T04:00:43.839ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 511fa71f-9d49-431d-90d0-c34f01fc4ef8 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "f21d5dfe-4022-49e6-a784-1e17499c3502", "content-length": "682", "date": "Tue, 25 Feb 2025 04:00:43 GMT"} })
7542025-02-25T04:00:43.839ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
7552025-02-25T04:00:43.840ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
7562025-02-25T04:00:43.843ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
7572025-02-25T04:00:43.843ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
7582025-02-25T04:00:43.847ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
7592025-02-25T04:00:43.847ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
7602025-02-25T04:00:43.849ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
7612025-02-25T04:00:43.849ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
7622025-02-25T04:00:43.856ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 710.603822ms iteration = 25 status = {}
7632025-02-25T04:00:45.635ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.15.1/src/server.rs:867 latency_us = 347 local_addr = [fd00:1122:3344:101::c]:52384 method = GET remote_addr = [fd00:1122:3344:101::d]:48316 req_id = 21061458-0c12-45c1-a7e2-176fb0f0bed3 response_code = 200 uri = /d8b217e1-a0a4-419c-82c9-30f4d54f8fbd
7642025-02-25T04:00:53.144ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 74 reason = Timeout
7652025-02-25T04:00:53.144ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 74 reason = Timeout
7662025-02-25T04:00:53.144ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
7672025-02-25T04:00:53.144ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7682025-02-25T04:00:53.144ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7692025-02-25T04:00:53.144ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
7702025-02-25T04:00:53.146ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7712025-02-25T04:00:53.149ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
7722025-02-25T04:00:53.149ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7732025-02-25T04:00:53.149ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7742025-02-25T04:00:53.151ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7752025-02-25T04:00:53.151ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7762025-02-25T04:00:53.152ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7772025-02-25T04:00:53.154ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 84829298-e6df-4d09-ba9a-550ee8541f9e, lookup_type: ById(84829298-e6df-4d09-ba9a-550ee8541f9e) } result = Ok(())
7782025-02-25T04:00:53.154ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7792025-02-25T04:00:53.154ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7802025-02-25T04:00:53.164ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 84829298-e6df-4d09-ba9a-550ee8541f9e original_time_created = 2025-02-25 03:48:33.610641 UTC target_id = 84829298-e6df-4d09-ba9a-550ee8541f9e time_created = 2025-02-25 03:48:33.610641 UTC
7812025-02-25T04:00:53.164ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 19.599708ms iteration = 74 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"84829298-e6df-4d09-ba9a-550ee8541f9e","time_created":"2025-02-25T03:48:33.610641Z"}
7822025-02-25T04:00:53.175ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
7832025-02-25T04:00:53.175ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
7842025-02-25T04:00:53.176ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 31.966415ms iteration = 74 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}