Max level shown:
12025-03-07T22:45:04.412ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 68 reason = Timeout
22025-03-07T22:45:04.412ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 68 reason = Timeout
32025-03-07T22:45:04.412ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
42025-03-07T22:45:04.413ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
52025-03-07T22:45:04.413ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
62025-03-07T22:45:04.413ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
72025-03-07T22:45:04.414ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
82025-03-07T22:45:04.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
92025-03-07T22:45:04.415ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102025-03-07T22:45:04.417ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.128289ms iteration = 68 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
112025-03-07T22:45:04.417ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
122025-03-07T22:45:04.417ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
132025-03-07T22:45:04.418ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
142025-03-07T22:45:04.420ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
152025-03-07T22:45:04.420ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
162025-03-07T22:45:04.421ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
172025-03-07T22:45:04.423ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
182025-03-07T22:45:04.423ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
192025-03-07T22:45:04.423ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
202025-03-07T22:45:04.433ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
212025-03-07T22:45:04.433ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.212355ms iteration = 68 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
222025-03-07T22:45:04.671ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:51000 req_id = d4783b48-f304-493c-aada-4ba506d5c6c4 roles = RoleSet { roles: {} } uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?sort_by=id_ascending
232025-03-07T22:45:04.672ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:51000 req_id = d4783b48-f304-493c-aada-4ba506d5c6c4 resource = Database result = Ok(()) uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?sort_by=id_ascending
242025-03-07T22:45:04.674ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3096 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:51000 req_id = d4783b48-f304-493c-aada-4ba506d5c6c4 response_code = 200 uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?sort_by=id_ascending
252025-03-07T22:45:04.675ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:51000 req_id = e4a597a2-a7b6-4a90-a42a-df39852c00bb roles = RoleSet { roles: {} } uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZmUxNjlhMy03NTM2LTRiNTQtOTU5My04ODE1ZDZlODFkZDQifX0%3D
262025-03-07T22:45:04.675ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:51000 req_id = e4a597a2-a7b6-4a90-a42a-df39852c00bb resource = Database result = Ok(()) uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZmUxNjlhMy03NTM2LTRiNTQtOTU5My04ODE1ZDZlODFkZDQifX0%3D
272025-03-07T22:45:04.677ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2217 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:51000 req_id = e4a597a2-a7b6-4a90-a42a-df39852c00bb response_code = 200 uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZmUxNjlhMy03NTM2LTRiNTQtOTU5My04ODE1ZDZlODFkZDQifX0%3D
282025-03-07T22:45:04.678ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 383 local_addr = [fd00:1122:3344:101::c]:41480 method = GET remote_addr = [fd00:1122:3344:101::d]:63523 req_id = 8fc95544-c588-4628-9ab1-c6039abd7867 response_code = 200 uri = /c0f69897-7f92-43db-ad47-96c3c4721128
292025-03-07T22:45:14.413ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 69 reason = Timeout
302025-03-07T22:45:14.413ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 69 reason = Timeout
312025-03-07T22:45:14.413ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
322025-03-07T22:45:14.413ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
332025-03-07T22:45:14.413ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
342025-03-07T22:45:14.413ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
352025-03-07T22:45:14.415ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
362025-03-07T22:45:14.418ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
372025-03-07T22:45:14.418ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
382025-03-07T22:45:14.418ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
392025-03-07T22:45:14.420ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
402025-03-07T22:45:14.420ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
412025-03-07T22:45:14.422ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
422025-03-07T22:45:14.426ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
432025-03-07T22:45:14.426ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
442025-03-07T22:45:14.426ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
452025-03-07T22:45:14.441ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
462025-03-07T22:45:14.441ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 28.640885ms iteration = 69 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
472025-03-07T22:45:14.452ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
482025-03-07T22:45:14.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
492025-03-07T22:45:14.454ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 41.386568ms iteration = 69 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
502025-03-07T22:45:14.679ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 467 local_addr = [fd00:1122:3344:101::c]:41480 method = GET remote_addr = [fd00:1122:3344:101::d]:63523 req_id = b713d102-ea4c-4fb3-a11c-9a66a034bc14 response_code = 200 uri = /c0f69897-7f92-43db-ad47-96c3c4721128
512025-03-07T22:45:24.412ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 26 reason = Timeout
522025-03-07T22:45:24.413ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
532025-03-07T22:45:24.413ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 24 reason = Timeout
542025-03-07T22:45:24.413ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 24 reason = Timeout
552025-03-07T22:45:24.413ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
562025-03-07T22:45:24.413ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
572025-03-07T22:45:24.413ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
582025-03-07T22:45:24.413ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 24 reason = Timeout
592025-03-07T22:45:24.413ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
602025-03-07T22:45:24.413ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 24 reason = Timeout
612025-03-07T22:45:24.413ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 24 reason = Timeout
622025-03-07T22:45:24.413ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
632025-03-07T22:45:24.413ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 24 reason = Timeout
642025-03-07T22:45:24.413ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
652025-03-07T22:45:24.413ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
662025-03-07T22:45:24.413ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
672025-03-07T22:45:24.413ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 24 reason = Timeout
682025-03-07T22:45:24.413ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
692025-03-07T22:45:24.413ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 24 reason = Timeout
702025-03-07T22:45:24.413ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
712025-03-07T22:45:24.413ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 70 reason = Timeout
722025-03-07T22:45:24.413ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 24 reason = Timeout
732025-03-07T22:45:24.413ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
742025-03-07T22:45:24.413ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
752025-03-07T22:45:24.413ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 24 reason = Timeout
762025-03-07T22:45:24.413ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
772025-03-07T22:45:24.413ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 70 reason = Timeout
782025-03-07T22:45:24.413ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
792025-03-07T22:45:24.413ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 24 reason = Timeout
802025-03-07T22:45:24.413ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
812025-03-07T22:45:24.413ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 24 reason = Timeout
822025-03-07T22:45:24.413ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
832025-03-07T22:45:24.413ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
842025-03-07T22:45:24.413ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
852025-03-07T22:45:24.413ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
862025-03-07T22:45:24.413ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
872025-03-07T22:45:24.413ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
882025-03-07T22:45:24.413ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
892025-03-07T22:45:24.413ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
902025-03-07T22:45:24.413ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
912025-03-07T22:45:24.413ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
922025-03-07T22:45:24.413ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
932025-03-07T22:45:24.413ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
942025-03-07T22:45:24.413ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
952025-03-07T22:45:24.413ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
962025-03-07T22:45:24.414ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 24 reason = Timeout
972025-03-07T22:45:24.414ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 24 reason = Timeout
982025-03-07T22:45:24.414ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 24 reason = Timeout
992025-03-07T22:45:24.414ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
1002025-03-07T22:45:24.414ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
1012025-03-07T22:45:24.414ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
1022025-03-07T22:45:24.414ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 24 reason = Timeout
1032025-03-07T22:45:24.414ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
1042025-03-07T22:45:24.414ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
1052025-03-07T22:45:24.414ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
1062025-03-07T22:45:24.414ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
1072025-03-07T22:45:24.414ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
1082025-03-07T22:45:24.417ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
1092025-03-07T22:45:24.417ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
1102025-03-07T22:45:24.417ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
1112025-03-07T22:45:24.417ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
1122025-03-07T22:45:24.422ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1132025-03-07T22:45:24.425ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
1142025-03-07T22:45:24.425ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
1152025-03-07T22:45:24.425ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
1162025-03-07T22:45:24.426ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1172025-03-07T22:45:24.427ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1182025-03-07T22:45:24.428ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1482, tv_nsec: 775373620 } })
1192025-03-07T22:45:24.429ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
1202025-03-07T22:45:24.429ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
1212025-03-07T22:45:24.429ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
1222025-03-07T22:45:24.429ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
1232025-03-07T22:45:24.429ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1242025-03-07T22:45:24.430ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
1252025-03-07T22:45:24.430ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1262025-03-07T22:45:24.433ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1272025-03-07T22:45:24.433ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1282025-03-07T22:45:24.433ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1482, tv_nsec: 780162906 } })
1292025-03-07T22:45:24.433ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
1302025-03-07T22:45:24.433ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1312025-03-07T22:45:24.435ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
1322025-03-07T22:45:24.435ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
1332025-03-07T22:45:24.436ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
1342025-03-07T22:45:24.436ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
1352025-03-07T22:45:24.436ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
1362025-03-07T22:45:24.436ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
1372025-03-07T22:45:24.436ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
1382025-03-07T22:45:24.436ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
1392025-03-07T22:45:24.436ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
1402025-03-07T22:45:24.440ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
1412025-03-07T22:45:24.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
1422025-03-07T22:45:24.441ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1432025-03-07T22:45:24.441ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1442025-03-07T22:45:24.441ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1452025-03-07T22:45:24.442ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1462025-03-07T22:45:24.442ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1482, tv_nsec: 789175972 } })
1472025-03-07T22:45:24.442ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
1482025-03-07T22:45:24.443ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
1492025-03-07T22:45:24.443ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
1502025-03-07T22:45:24.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
1512025-03-07T22:45:24.444ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
1522025-03-07T22:45:24.444ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
1532025-03-07T22:45:24.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
1542025-03-07T22:45:24.445ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 32.085248ms iteration = 24 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
1552025-03-07T22:45:24.476ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1562025-03-07T22:45:24.476ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1572025-03-07T22:45:24.481ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1582025-03-07T22:45:24.484ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
1592025-03-07T22:45:24.484ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
1602025-03-07T22:45:24.484ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
1612025-03-07T22:45:24.485ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 71.390596ms iteration = 24 status = {"errors":[],"finish_invoked_ok":[]}
1622025-03-07T22:45:24.487ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
1632025-03-07T22:45:24.487ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
1642025-03-07T22:45:24.487ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
1652025-03-07T22:45:24.488ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
1662025-03-07T22:45:24.491ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 77.580256ms iteration = 24 status = {"errors":[],"garbage_collect_requested":[]}
1672025-03-07T22:45:24.494ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
1682025-03-07T22:45:24.494ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
1692025-03-07T22:45:24.495ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
1702025-03-07T22:45:24.495ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
1712025-03-07T22:45:24.497ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
1722025-03-07T22:45:24.497ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
1732025-03-07T22:45:24.497ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 83.452072ms iteration = 24 status = {"errors":[],"requests_created_ok":[]}
1742025-03-07T22:45:24.497ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
1752025-03-07T22:45:24.497ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
1762025-03-07T22:45:24.498ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "218c4d61-51c0-439f-8a02-10ebbb24f8cb", "content-length": "26", "date": "Fri, 07 Mar 2025 22:45:24 GMT"} })
1772025-03-07T22:45:24.498ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
1782025-03-07T22:45:24.499ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
1792025-03-07T22:45:24.505ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
1802025-03-07T22:45:24.505ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
1812025-03-07T22:45:24.505ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
1822025-03-07T22:45:24.506ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
1832025-03-07T22:45:24.509ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
1842025-03-07T22:45:24.509ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
1852025-03-07T22:45:24.509ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "dfcf0584-a5c6-4fb9-bdfa-200fa917348a", "content-length": "26", "date": "Fri, 07 Mar 2025 22:45:24 GMT"} })
1862025-03-07T22:45:24.509ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
1872025-03-07T22:45:24.511ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
1882025-03-07T22:45:24.534ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
1892025-03-07T22:45:24.535ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
1902025-03-07T22:45:24.536ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
1912025-03-07T22:45:24.536ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
1922025-03-07T22:45:24.537ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7070ead-b325-4d4c-9a91-c507f114063c", "content-length": "583", "date": "Fri, 07 Mar 2025 22:45:24 GMT"} })
1932025-03-07T22:45:24.537ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
1942025-03-07T22:45:24.538ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
1952025-03-07T22:45:24.539ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1962025-03-07T22:45:24.539ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1972025-03-07T22:45:24.543ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec zone_address = fd00:1122:3344:101::2
1982025-03-07T22:45:24.543ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
1992025-03-07T22:45:24.545ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "74261bc7-6121-47e3-85b7-ffba5d4a0be0", "content-length": "26", "date": "Fri, 07 Mar 2025 22:45:24 GMT"} })
2002025-03-07T22:45:24.545ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
2012025-03-07T22:45:24.547ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
2022025-03-07T22:45:24.563ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 150.997591ms iteration = 26 status = {"physical_disks_added":0}
2032025-03-07T22:45:24.612ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = instance_watcher body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vmms/03cf82ac-c6eb-49ea-9515-bc937b5eb030/state
2042025-03-07T22:45:24.614ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
2052025-03-07T22:45:24.614ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "789c4d3a-41c5-48a5-99b9-f4392b0c8223", "content-length": "1", "date": "Fri, 07 Mar 2025 22:45:24 GMT"} })
2062025-03-07T22:45:24.614ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2072025-03-07T22:45:24.615ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2082025-03-07T22:45:24.615ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
2092025-03-07T22:45:24.615ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = instance_watcher result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vmms/03cf82ac-c6eb-49ea-9515-bc937b5eb030/state", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf0ee30a-d10d-4368-be5e-288f69809075", "content-length": "130", "date": "Fri, 07 Mar 2025 22:45:24 GMT"} })
2102025-03-07T22:45:24.615ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
2112025-03-07T22:45:24.620ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 207.556035ms iteration = 24 status = null
2122025-03-07T22:45:24.622ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 208.982409ms iteration = 24 status = {}
2132025-03-07T22:45:24.628ZDEBGnexus (BackgroundTasks): updating instance state background_task = instance_watcher instance_id = 831cf520-76c6-431b-a34a-a29e6222f280 sled_id = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 state = SledVmmState { vmm_state: VmmRuntimeState { state: Running, gen: Generation(4), time_updated: 2025-03-07T22:45:16.429817750Z }, migration_in: None, migration_out: None } vmm_id = 03cf82ac-c6eb-49ea-9515-bc937b5eb030
2142025-03-07T22:45:24.628ZINFOnexus (BackgroundTasks): received new VMM runtime state from sled agent background_task = instance_watcher file = nexus/src/app/instance.rs:2088 instance_id = 831cf520-76c6-431b-a34a-a29e6222f280 migration_state = Migrations { migration_in: None, migration_out: None } propolis_id = 03cf82ac-c6eb-49ea-9515-bc937b5eb030 sled_id = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 vmm_id = 03cf82ac-c6eb-49ea-9515-bc937b5eb030 vmm_state = VmmRuntimeState { state: Running, gen: Generation(4), time_updated: 2025-03-07T22:45:16.429817750Z }
2152025-03-07T22:45:24.628ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher instance_id = 831cf520-76c6-431b-a34a-a29e6222f280 roles = RoleSet { roles: {} } sled_id = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 vmm_id = 03cf82ac-c6eb-49ea-9515-bc937b5eb030
2162025-03-07T22:45:24.629ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher instance_id = 831cf520-76c6-431b-a34a-a29e6222f280 resource = Database result = Ok(()) sled_id = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 vmm_id = 03cf82ac-c6eb-49ea-9515-bc937b5eb030
2172025-03-07T22:45:24.647ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
2182025-03-07T22:45:24.647ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
2192025-03-07T22:45:24.648ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 234.815786ms iteration = 70 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
2202025-03-07T22:45:24.648ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 235.042909ms iteration = 24 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
2212025-03-07T22:45:24.649ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2222025-03-07T22:45:24.649ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 236.788806ms iteration = 24 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
2232025-03-07T22:45:24.651ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
2242025-03-07T22:45:24.651ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
2252025-03-07T22:45:24.651ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 238.767325ms iteration = 24 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
2262025-03-07T22:45:24.651ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
2272025-03-07T22:45:24.651ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
2282025-03-07T22:45:24.652ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2292025-03-07T22:45:24.652ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
2302025-03-07T22:45:24.655ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2312025-03-07T22:45:24.655ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
2322025-03-07T22:45:24.655ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2332025-03-07T22:45:24.656ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2342025-03-07T22:45:24.658ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 73b218dd-7dbc-4345-acb1-fe69ced4a7f7, lookup_type: ById(73b218dd-7dbc-4345-acb1-fe69ced4a7f7) } result = Ok(())
2352025-03-07T22:45:24.658ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
2362025-03-07T22:45:24.658ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
2372025-03-07T22:45:24.661ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 248.241825ms iteration = 24 status = 0
2382025-03-07T22:45:24.663ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {be157122-2b34-4fd6-9919-1f578494e18f: {10.151.1.175: {3ada68f0-1bcd-4a51-8fdc-58e69fb3bda1}, 10.151.1.176: {3ada68f0-1bcd-4a51-8fdc-58e69fb3bda1}}} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
2392025-03-07T22:45:24.663ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
2402025-03-07T22:45:24.663ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "720d32cb-4f30-43e4-8eb4-cdce3c3bc66d", "date": "Fri, 07 Mar 2025 22:45:24 GMT"} })
2412025-03-07T22:45:24.663ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
2422025-03-07T22:45:24.664ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
2432025-03-07T22:45:24.666ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
2442025-03-07T22:45:24.666ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
2452025-03-07T22:45:24.669ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
2462025-03-07T22:45:24.669ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
2472025-03-07T22:45:24.670ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 1 total_failed = 0 total_incomplete = 0 total_instances = 1 update_sagas_queued = 0
2482025-03-07T22:45:24.671ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
2492025-03-07T22:45:24.671ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 258.717347ms iteration = 70 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
2502025-03-07T22:45:24.672ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 259.066799ms iteration = 24 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{"running":1},"pruned_instances":0,"total_instances":1,"update_sagas_queued":0}
2512025-03-07T22:45:24.672ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
2522025-03-07T22:45:24.672ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
2532025-03-07T22:45:24.674ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
2542025-03-07T22:45:24.674ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
2552025-03-07T22:45:24.677ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
2562025-03-07T22:45:24.677ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
2572025-03-07T22:45:24.679ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
2582025-03-07T22:45:24.679ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 265.123578ms iteration = 24 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
2592025-03-07T22:45:24.679ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
2602025-03-07T22:45:24.679ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
2612025-03-07T22:45:24.679ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "d3a9c50b-5c10-4994-9307-d579b1e81eee", "date": "Fri, 07 Mar 2025 22:45:24 GMT"} })
2622025-03-07T22:45:24.680ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 500 local_addr = [fd00:1122:3344:101::c]:41480 method = GET remote_addr = [fd00:1122:3344:101::d]:63523 req_id = 608f7c78-d9b8-4554-8c61-f30fc860222b response_code = 200 uri = /c0f69897-7f92-43db-ad47-96c3c4721128
2632025-03-07T22:45:24.682ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 269.317868ms iteration = 24 status = {}
2642025-03-07T22:45:24.686ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
2652025-03-07T22:45:24.686ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 273.60608ms iteration = 24 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
2662025-03-07T22:45:24.701ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
2672025-03-07T22:45:24.701ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
2682025-03-07T22:45:24.702ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 2619304f-a0b9-451a-bbfb-b14af6353df6, rack_id: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(8820d7be-fc72-45ff-bc2e-17c4bb9613af) } rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
2692025-03-07T22:45:24.702ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
2702025-03-07T22:45:24.702ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
2712025-03-07T22:45:24.710ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
2722025-03-07T22:45:24.711ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e7a2385-d14a-48f2-a3b4-54357e81963d", "content-length": "134", "date": "Fri, 07 Mar 2025 22:45:24 GMT"} })
2732025-03-07T22:45:24.711ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
2742025-03-07T22:45:24.711ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 8820d7be-fc72-45ff-bc2e-17c4bb9613af
2752025-03-07T22:45:24.711ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
2762025-03-07T22:45:24.711ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
2772025-03-07T22:45:24.711ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
2782025-03-07T22:45:24.711ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
2792025-03-07T22:45:24.711ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
2802025-03-07T22:45:24.711ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
2812025-03-07T22:45:24.711ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
2822025-03-07T22:45:24.711ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
2832025-03-07T22:45:24.711ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
2842025-03-07T22:45:24.711ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
2852025-03-07T22:45:24.712ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
2862025-03-07T22:45:24.714ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
2872025-03-07T22:45:24.714ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "68e733c4-eee9-45dd-b229-3ea46c0e2a1f", "content-length": "2", "date": "Fri, 07 Mar 2025 22:45:24 GMT"} })
2882025-03-07T22:45:24.714ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
2892025-03-07T22:45:24.714ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "634e9aa3-19e5-4605-a309-52bd028ccb65", "content-length": "52", "date": "Fri, 07 Mar 2025 22:45:24 GMT"} })
2902025-03-07T22:45:24.714ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
2912025-03-07T22:45:24.714ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
2922025-03-07T22:45:24.722ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
2932025-03-07T22:45:24.722ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
2942025-03-07T22:45:24.750ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2952025-03-07T22:45:24.752ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
2962025-03-07T22:45:24.752ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
2972025-03-07T22:45:24.752ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
2982025-03-07T22:45:24.781ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2992025-03-07T22:45:24.783ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
3002025-03-07T22:45:24.783ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
3012025-03-07T22:45:24.783ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
3022025-03-07T22:45:24.818ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "0f76bf66-cb7d-4cb8-ba86-750b9c551987", "date": "Fri, 07 Mar 2025 22:45:24 GMT"} })
3032025-03-07T22:45:24.818ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch = Switch0
3042025-03-07T22:45:24.818ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
3052025-03-07T22:45:24.819ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b80eea1-9816-490f-ac6c-0d1ee4fe28ad", "content-length": "682", "date": "Fri, 07 Mar 2025 22:45:24 GMT"} })
3062025-03-07T22:45:24.819ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
3072025-03-07T22:45:24.819ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
3082025-03-07T22:45:24.822ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
3092025-03-07T22:45:24.823ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
3102025-03-07T22:45:24.826ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
3112025-03-07T22:45:24.826ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
3122025-03-07T22:45:24.828ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
3132025-03-07T22:45:24.829ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
3142025-03-07T22:45:24.836ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 422.901234ms iteration = 24 status = {}
3152025-03-07T22:45:24.873ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
3162025-03-07T22:45:24.873ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1674f1c-0637-41f8-9cc4-67b6ef9d5fde", "content-length": "810", "date": "Fri, 07 Mar 2025 22:45:24 GMT"} })
3172025-03-07T22:45:24.873ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {VirtualNetworkInterfaceHost { physical_host_ip: fd00:1122:3344:101::1, virtual_ip: 172.30.0.5, virtual_mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), vni: Vni(2073255) }} sled = buskin
3182025-03-07T22:45:24.873ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
3192025-03-07T22:45:24.873ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
3202025-03-07T22:45:24.875ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 462.757237ms iteration = 24 status = {}
3212025-03-07T22:45:34.413ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 71 reason = Timeout
3222025-03-07T22:45:34.413ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 71 reason = Timeout
3232025-03-07T22:45:34.413ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3242025-03-07T22:45:34.413ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
3252025-03-07T22:45:34.413ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3262025-03-07T22:45:34.413ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
3272025-03-07T22:45:34.450ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
3282025-03-07T22:45:34.450ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
3292025-03-07T22:45:34.451ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3302025-03-07T22:45:34.452ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 39.176697ms iteration = 71 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
3312025-03-07T22:45:34.454ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
3322025-03-07T22:45:34.454ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3332025-03-07T22:45:34.454ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3342025-03-07T22:45:34.456ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3352025-03-07T22:45:34.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3362025-03-07T22:45:34.458ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3372025-03-07T22:45:34.460ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
3382025-03-07T22:45:34.460ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3392025-03-07T22:45:34.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3402025-03-07T22:45:34.471ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
3412025-03-07T22:45:34.471ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 58.852366ms iteration = 71 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
3422025-03-07T22:45:34.679ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 374 local_addr = [fd00:1122:3344:101::c]:41480 method = GET remote_addr = [fd00:1122:3344:101::d]:63523 req_id = 6bb23757-5cb4-48a4-a2c3-3d9d994dca15 response_code = 200 uri = /c0f69897-7f92-43db-ad47-96c3c4721128
3432025-03-07T22:45:44.413ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 72 reason = Timeout
3442025-03-07T22:45:44.413ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 72 reason = Timeout
3452025-03-07T22:45:44.413ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3462025-03-07T22:45:44.413ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
3472025-03-07T22:45:44.413ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
3482025-03-07T22:45:44.413ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3492025-03-07T22:45:44.415ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
3502025-03-07T22:45:44.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
3512025-03-07T22:45:44.416ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3522025-03-07T22:45:44.417ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.352912ms iteration = 72 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
3532025-03-07T22:45:44.419ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
3542025-03-07T22:45:44.419ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3552025-03-07T22:45:44.419ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3562025-03-07T22:45:44.421ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3572025-03-07T22:45:44.421ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3582025-03-07T22:45:44.423ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3592025-03-07T22:45:44.425ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
3602025-03-07T22:45:44.425ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3612025-03-07T22:45:44.426ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3622025-03-07T22:45:44.437ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
3632025-03-07T22:45:44.437ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 24.083101ms iteration = 72 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
3642025-03-07T22:45:44.679ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 547 local_addr = [fd00:1122:3344:101::c]:41480 method = GET remote_addr = [fd00:1122:3344:101::d]:63523 req_id = cd5e389c-5b73-4ae5-b1b4-8227797f6aad response_code = 200 uri = /c0f69897-7f92-43db-ad47-96c3c4721128
3652025-03-07T22:45:54.413ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 27 reason = Timeout
3662025-03-07T22:45:54.413ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
3672025-03-07T22:45:54.413ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 14 reason = Timeout
3682025-03-07T22:45:54.413ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f
3692025-03-07T22:45:54.413ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 25 reason = Timeout
3702025-03-07T22:45:54.413ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 17.5\u{b5}s iteration = 14 status = {"enabled":false,"target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f"}
3712025-03-07T22:45:54.413ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
3722025-03-07T22:45:54.413ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 13 reason = Timeout
3732025-03-07T22:45:54.413ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 25 reason = Timeout
3742025-03-07T22:45:54.413ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
3752025-03-07T22:45:54.413ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
3762025-03-07T22:45:54.413ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
3772025-03-07T22:45:54.413ZDEBGnexus (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 6 reason = Timeout
3782025-03-07T22:45:54.413ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
3792025-03-07T22:45:54.413ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
3802025-03-07T22:45:54.413ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 13 reason = Timeout
3812025-03-07T22:45:54.413ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
3822025-03-07T22:45:54.413ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 15 reason = Timeout
3832025-03-07T22:45:54.413ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 25 reason = Timeout
3842025-03-07T22:45:54.413ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 73 reason = Timeout
3852025-03-07T22:45:54.413ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
3862025-03-07T22:45:54.413ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
3872025-03-07T22:45:54.413ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 25 reason = Timeout
3882025-03-07T22:45:54.413ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 25 reason = Timeout
3892025-03-07T22:45:54.413ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
3902025-03-07T22:45:54.413ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
3912025-03-07T22:45:54.413ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 73 reason = Timeout
3922025-03-07T22:45:54.413ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3932025-03-07T22:45:54.413ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 25 reason = Timeout
3942025-03-07T22:45:54.413ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 25 reason = Timeout
3952025-03-07T22:45:54.413ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
3962025-03-07T22:45:54.413ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 25 reason = Timeout
3972025-03-07T22:45:54.413ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
3982025-03-07T22:45:54.413ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 13 reason = Timeout
3992025-03-07T22:45:54.413ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 25 reason = Timeout
4002025-03-07T22:45:54.413ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-07 22:35:54.413570748 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
4012025-03-07T22:45:54.413ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
4022025-03-07T22:45:54.413ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 13 reason = Timeout
4032025-03-07T22:45:54.413ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
4042025-03-07T22:45:54.413ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 15 reason = Timeout
4052025-03-07T22:45:54.413ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
4062025-03-07T22:45:54.413ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 13 reason = Timeout
4072025-03-07T22:45:54.413ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 25 reason = Timeout
4082025-03-07T22:45:54.413ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
4092025-03-07T22:45:54.413ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 13 reason = Timeout
4102025-03-07T22:45:54.413ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
4112025-03-07T22:45:54.413ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
4122025-03-07T22:45:54.413ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 13 reason = Timeout
4132025-03-07T22:45:54.413ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
4142025-03-07T22:45:54.413ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
4152025-03-07T22:45:54.413ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 13 reason = Timeout
4162025-03-07T22:45:54.413ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
4172025-03-07T22:45:54.413ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 25 reason = Timeout
4182025-03-07T22:45:54.414ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 13 reason = Timeout
4192025-03-07T22:45:54.414ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
4202025-03-07T22:45:54.414ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
4212025-03-07T22:45:54.414ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 13 reason = Timeout
4222025-03-07T22:45:54.414ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
4232025-03-07T22:45:54.414ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 25 reason = Timeout
4242025-03-07T22:45:54.414ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
4252025-03-07T22:45:54.414ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
4262025-03-07T22:45:54.414ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
4272025-03-07T22:45:54.414ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
4282025-03-07T22:45:54.414ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
4292025-03-07T22:45:54.414ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
4302025-03-07T22:45:54.414ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
4312025-03-07T22:45:54.414ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
4322025-03-07T22:45:54.414ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
4332025-03-07T22:45:54.414ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4342025-03-07T22:45:54.414ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
4352025-03-07T22:45:54.414ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
4362025-03-07T22:45:54.414ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
4372025-03-07T22:45:54.414ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
4382025-03-07T22:45:54.414ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
4392025-03-07T22:45:54.414ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
4402025-03-07T22:45:54.414ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 25 reason = Timeout
4412025-03-07T22:45:54.414ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 25 reason = Timeout
4422025-03-07T22:45:54.414ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 25 reason = Timeout
4432025-03-07T22:45:54.414ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
4442025-03-07T22:45:54.414ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
4452025-03-07T22:45:54.414ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
4462025-03-07T22:45:54.414ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 25 reason = Timeout
4472025-03-07T22:45:54.414ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
4482025-03-07T22:45:54.414ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
4492025-03-07T22:45:54.414ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
4502025-03-07T22:45:54.414ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
4512025-03-07T22:45:54.414ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
4522025-03-07T22:45:54.414ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
4532025-03-07T22:45:54.414ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
4542025-03-07T22:45:54.414ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
4552025-03-07T22:45:54.414ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
4562025-03-07T22:45:54.414ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
4572025-03-07T22:45:54.414ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
4582025-03-07T22:45:54.414ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("732b0e55-5ccf-4068-b771-1299f872f618.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("fbf58a59-609b-4fb3-b792-2c09f3f7b776.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("fbf58a59-609b-4fb3-b792-2c09f3f7b776.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 1512, tv_nsec: 760330705 } })
4592025-03-07T22:45:54.414ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
4602025-03-07T22:45:54.414ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("114e494f-6e94-4e0c-80db-5823b226f1a3.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("a9773e0b-15d4-4efa-b977-9f3598685cdd.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("d2f04f38-7ac2-4327-a087-c2a47865841a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("d2f04f38-7ac2-4327-a087-c2a47865841a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 1512, tv_nsec: 760351995 } })
4612025-03-07T22:45:54.414ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
4622025-03-07T22:45:54.414ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
4632025-03-07T22:45:54.414ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
4642025-03-07T22:45:54.414ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
4652025-03-07T22:45:54.414ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
4662025-03-07T22:45:54.414ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
4672025-03-07T22:45:54.414ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
4682025-03-07T22:45:54.414ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
4692025-03-07T22:45:54.414ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
4702025-03-07T22:45:54.414ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.11466ms iteration = 13 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
4712025-03-07T22:45:54.414ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
4722025-03-07T22:45:54.415ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
4732025-03-07T22:45:54.415ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.392883ms iteration = 13 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353"]}
4742025-03-07T22:45:54.415ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
4752025-03-07T22:45:54.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
4762025-03-07T22:45:54.416ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4772025-03-07T22:45:54.417ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4782025-03-07T22:45:54.417ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1512, tv_nsec: 763736698 } })
4792025-03-07T22:45:54.417ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.444553ms iteration = 25 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
4802025-03-07T22:45:54.418ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
4812025-03-07T22:45:54.418ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
4822025-03-07T22:45:54.418ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
4832025-03-07T22:45:54.418ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
4842025-03-07T22:45:54.419ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
4852025-03-07T22:45:54.419ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
4862025-03-07T22:45:54.419ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
4872025-03-07T22:45:54.421ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
4882025-03-07T22:45:54.421ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
4892025-03-07T22:45:54.422ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4902025-03-07T22:45:54.422ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4912025-03-07T22:45:54.428ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
4922025-03-07T22:45:54.428ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
4932025-03-07T22:45:54.428ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
4942025-03-07T22:45:54.428ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
4952025-03-07T22:45:54.428ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
4962025-03-07T22:45:54.428ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
4972025-03-07T22:45:54.428ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
4982025-03-07T22:45:54.428ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
4992025-03-07T22:45:54.430ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1512, tv_nsec: 776704062 } })
5002025-03-07T22:45:54.431ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
5012025-03-07T22:45:54.432ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-07 22:35:54.413570748 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
5022025-03-07T22:45:54.432ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 18.800901ms iteration = 13 status = {"expiration":"2025-03-07T22:35:54.413570748Z","pruned":[]}
5032025-03-07T22:45:54.434ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 20.690469ms iteration = 13 status = {"errors":[],"found_port_ok":[]}
5042025-03-07T22:45:54.473ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
5052025-03-07T22:45:54.473ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
5062025-03-07T22:45:54.473ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
5072025-03-07T22:45:54.473ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
5082025-03-07T22:45:54.474ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
5092025-03-07T22:45:54.474ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5102025-03-07T22:45:54.474ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
5112025-03-07T22:45:54.474ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5122025-03-07T22:45:54.476ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
5132025-03-07T22:45:54.476ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
5142025-03-07T22:45:54.476ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
5152025-03-07T22:45:54.477ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
5162025-03-07T22:45:54.477ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
5172025-03-07T22:45:54.477ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 63.911854ms iteration = 13 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
5182025-03-07T22:45:54.478ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5192025-03-07T22:45:54.479ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
5202025-03-07T22:45:54.479ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
5212025-03-07T22:45:54.479ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
5222025-03-07T22:45:54.481ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5232025-03-07T22:45:54.482ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
5242025-03-07T22:45:54.482ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
5252025-03-07T22:45:54.482ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
5262025-03-07T22:45:54.483ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
5272025-03-07T22:45:54.483ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
5282025-03-07T22:45:54.483ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 69.302696ms iteration = 25 status = {"errors":[],"requests_created_ok":[]}
5292025-03-07T22:45:54.484ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
5302025-03-07T22:45:54.484ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
5312025-03-07T22:45:54.484ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
5322025-03-07T22:45:54.484ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
5332025-03-07T22:45:54.484ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
5342025-03-07T22:45:54.486ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
5352025-03-07T22:45:54.486ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
5362025-03-07T22:45:54.487ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5372025-03-07T22:45:54.487ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5382025-03-07T22:45:54.487ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5392025-03-07T22:45:54.487ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5402025-03-07T22:45:54.490ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5412025-03-07T22:45:54.490ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
5422025-03-07T22:45:54.490ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
5432025-03-07T22:45:54.490ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
5442025-03-07T22:45:54.491ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
5452025-03-07T22:45:54.491ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
5462025-03-07T22:45:54.492ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
5472025-03-07T22:45:54.492ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5482025-03-07T22:45:54.492ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
5492025-03-07T22:45:54.492ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
5502025-03-07T22:45:54.492ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
5512025-03-07T22:45:54.494ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
5522025-03-07T22:45:54.494ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 80.371313ms iteration = 25 status = {"errors":[],"garbage_collect_requested":[]}
5532025-03-07T22:45:54.494ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
5542025-03-07T22:45:54.494ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
5552025-03-07T22:45:54.494ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
5562025-03-07T22:45:54.494ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 80.722946ms iteration = 25 status = {"errors":[],"finish_invoked_ok":[]}
5572025-03-07T22:45:54.494ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
5582025-03-07T22:45:54.496ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5592025-03-07T22:45:54.497ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 84.214059ms iteration = 25 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
5602025-03-07T22:45:54.502ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
5612025-03-07T22:45:54.502ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
5622025-03-07T22:45:54.502ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
5632025-03-07T22:45:54.504ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
5642025-03-07T22:45:54.504ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5652025-03-07T22:45:54.504ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
5662025-03-07T22:45:54.504ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1512, tv_nsec: 850370270 } })
5672025-03-07T22:45:54.504ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
5682025-03-07T22:45:54.507ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
5692025-03-07T22:45:54.507ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
5702025-03-07T22:45:54.507ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5712025-03-07T22:45:54.507ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
5722025-03-07T22:45:54.510ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
5732025-03-07T22:45:54.510ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
5742025-03-07T22:45:54.511ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
5752025-03-07T22:45:54.512ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5762025-03-07T22:45:54.514ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
5772025-03-07T22:45:54.514ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
5782025-03-07T22:45:54.514ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
5792025-03-07T22:45:54.515ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5802025-03-07T22:45:54.518ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
5812025-03-07T22:45:54.518ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
5822025-03-07T22:45:54.518ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
5832025-03-07T22:45:54.521ZDEBGnexus (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 107.737206ms iteration = 6 status = {"nsuccess":5}
5842025-03-07T22:45:54.522ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
5852025-03-07T22:45:54.522ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
5862025-03-07T22:45:54.522ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
5872025-03-07T22:45:54.522ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
5882025-03-07T22:45:54.523ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5892025-03-07T22:45:54.525ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
5902025-03-07T22:45:54.525ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
5912025-03-07T22:45:54.526ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
5922025-03-07T22:45:54.526ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 113.014046ms iteration = 73 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
5932025-03-07T22:45:54.526ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5942025-03-07T22:45:54.528ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
5952025-03-07T22:45:54.529ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 115.50215ms iteration = 73 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
5962025-03-07T22:45:54.530ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
5972025-03-07T22:45:54.531ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5982025-03-07T22:45:54.532ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5992025-03-07T22:45:54.532ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 1 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
6002025-03-07T22:45:54.532ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 118.485508ms iteration = 25 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":1,"total_instances":0,"update_sagas_queued":0}
6012025-03-07T22:45:54.536ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
6022025-03-07T22:45:54.536ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
6032025-03-07T22:45:54.537ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6042025-03-07T22:45:54.537ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6052025-03-07T22:45:54.537ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
6062025-03-07T22:45:54.538ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
6072025-03-07T22:45:54.540ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
6082025-03-07T22:45:54.541ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
6092025-03-07T22:45:54.541ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 73b218dd-7dbc-4345-acb1-fe69ced4a7f7, lookup_type: ById(73b218dd-7dbc-4345-acb1-fe69ced4a7f7) } result = Ok(())
6102025-03-07T22:45:54.541ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
6112025-03-07T22:45:54.541ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
6122025-03-07T22:45:54.541ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
6132025-03-07T22:45:54.542ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
6142025-03-07T22:45:54.543ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
6152025-03-07T22:45:54.543ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 130.106941ms iteration = 25 status = 0
6162025-03-07T22:45:54.545ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6172025-03-07T22:45:54.545ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 132.253511ms iteration = 25 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
6182025-03-07T22:45:54.547ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
6192025-03-07T22:45:54.547ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
6202025-03-07T22:45:54.547ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
6212025-03-07T22:45:54.548ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
6222025-03-07T22:45:54.548ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
6232025-03-07T22:45:54.549ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
6242025-03-07T22:45:54.549ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 135.31501ms iteration = 13 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
6252025-03-07T22:45:54.555ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
6262025-03-07T22:45:54.555ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
6272025-03-07T22:45:54.555ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "032bb12e-24c4-4862-9e05-01363768bbb8", "content-length": "26", "date": "Fri, 07 Mar 2025 22:45:54 GMT"} })
6282025-03-07T22:45:54.555ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
6292025-03-07T22:45:54.557ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
6302025-03-07T22:45:54.568ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
6312025-03-07T22:45:54.568ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
6322025-03-07T22:45:54.570ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
6332025-03-07T22:45:54.570ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
6342025-03-07T22:45:54.570ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 157.22988ms iteration = 25 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
6352025-03-07T22:45:54.572ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
6362025-03-07T22:45:54.572ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
6372025-03-07T22:45:54.572ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
6382025-03-07T22:45:54.572ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
6392025-03-07T22:45:54.573ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
6402025-03-07T22:45:54.573ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6412025-03-07T22:45:54.575ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6422025-03-07T22:45:54.576ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
6432025-03-07T22:45:54.576ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
6442025-03-07T22:45:54.576ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
6452025-03-07T22:45:54.579ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
6462025-03-07T22:45:54.579ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
6472025-03-07T22:45:54.579ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
6482025-03-07T22:45:54.583ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
6492025-03-07T22:45:54.583ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
6502025-03-07T22:45:54.583ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
6512025-03-07T22:45:54.583ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
6522025-03-07T22:45:54.583ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
6532025-03-07T22:45:54.585ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
6542025-03-07T22:45:54.585ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6552025-03-07T22:45:54.585ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 171.762191ms iteration = 25 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
6562025-03-07T22:45:54.586ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
6572025-03-07T22:45:54.586ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 172.69605ms iteration = 13 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
6582025-03-07T22:45:54.587ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
6592025-03-07T22:45:54.587ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 173.80978ms iteration = 25 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
6602025-03-07T22:45:54.588ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
6612025-03-07T22:45:54.588ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
6622025-03-07T22:45:54.590ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
6632025-03-07T22:45:54.590ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
6642025-03-07T22:45:54.590ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "de786208-7fda-4597-a229-1ecf119b6647", "content-length": "26", "date": "Fri, 07 Mar 2025 22:45:54 GMT"} })
6652025-03-07T22:45:54.590ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
6662025-03-07T22:45:54.590ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
6672025-03-07T22:45:54.590ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
6682025-03-07T22:45:54.590ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
6692025-03-07T22:45:54.592ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
6702025-03-07T22:45:54.592ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
6712025-03-07T22:45:54.592ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = fdc30f7a-36a7-4893-91d6-3456da7362c0 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
6722025-03-07T22:45:54.592ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
6732025-03-07T22:45:54.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
6742025-03-07T22:45:54.621ZINFOnexus (BackgroundTasks): found Silo/DNS/TLS config (changed) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-07T22:33:52.171499Z, time_modified: 2025-03-07T22:33:52.171499Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, db_silo: Silo { identity: SiloIdentity { id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-07T22:34:17.695388Z, time_modified: 2025-03-07T22:34:17.695388Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, db_silo: Silo { identity: SiloIdentity { id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-07T22:34:17.695388Z, time_modified: 2025-03-07T22:34:17.695388Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1" }] }) } file = nexus/src/app/background/tasks/external_endpoints.rs:94
6752025-03-07T22:45:54.621ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 207.476124ms iteration = 13 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"ebb5815c-a32b-49c0-a621-3a7cd3eb3452","tls_certs":["f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1"]}},"default_endpoint":{"silo_id":"ebb5815c-a32b-49c0-a621-3a7cd3eb3452","tls_certs":["f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
6762025-03-07T22:45:54.623ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
6772025-03-07T22:45:54.623ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = a49f17b2-1834-4c40-bc6e-ff0588e4b3ef dns_zone_name = oxide.test found_so_far = 0 version = 2
6782025-03-07T22:45:54.623ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
6792025-03-07T22:45:54.623ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
6802025-03-07T22:45:54.625ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6812025-03-07T22:45:54.628ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
6822025-03-07T22:45:54.628ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
6832025-03-07T22:45:54.628ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
6842025-03-07T22:45:54.630ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = a49f17b2-1834-4c40-bc6e-ff0588e4b3ef dns_zone_name = oxide.test found_so_far = 1 version = 2
6852025-03-07T22:45:54.630ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
6862025-03-07T22:45:54.630ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-07 22:34:17.797135 UTC dns_group = external generation = 2
6872025-03-07T22:45:54.630ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 217.025946ms iteration = 13 status = {"generation":2}
6882025-03-07T22:45:54.632ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 218.938584ms iteration = 27 status = {"physical_disks_added":0}
6892025-03-07T22:45:54.637ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6902025-03-07T22:45:54.640ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
6912025-03-07T22:45:54.640ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
6922025-03-07T22:45:54.640ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
6932025-03-07T22:45:54.642ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = fdc30f7a-36a7-4893-91d6-3456da7362c0 dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
6942025-03-07T22:45:54.642ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
6952025-03-07T22:45:54.642ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-07 22:34:17.208516 UTC dns_group = internal generation = 1
6962025-03-07T22:45:54.642ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 229.483516ms iteration = 13 status = {"generation":1}
6972025-03-07T22:45:54.668ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec zone_address = fd00:1122:3344:101::2
6982025-03-07T22:45:54.668ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
6992025-03-07T22:45:54.669ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea738398-35fb-4b0b-ab85-8e9155f873c1", "content-length": "26", "date": "Fri, 07 Mar 2025 22:45:54 GMT"} })
7002025-03-07T22:45:54.669ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
7012025-03-07T22:45:54.670ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
7022025-03-07T22:45:54.679ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 328 local_addr = [fd00:1122:3344:101::c]:41480 method = GET remote_addr = [fd00:1122:3344:101::d]:63523 req_id = 380769a4-358e-4f28-bb7a-461c56683b16 response_code = 200 uri = /c0f69897-7f92-43db-ad47-96c3c4721128
7032025-03-07T22:45:54.682ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
7042025-03-07T22:45:54.683ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
7052025-03-07T22:45:54.685ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7062025-03-07T22:45:54.687ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
7072025-03-07T22:45:54.687ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
7082025-03-07T22:45:54.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
7092025-03-07T22:45:54.688ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7102025-03-07T22:45:54.690ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
7112025-03-07T22:45:54.690ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
7122025-03-07T22:45:54.691ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
7132025-03-07T22:45:54.696ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
7142025-03-07T22:45:54.711ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
7152025-03-07T22:45:54.711ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5f466d8-1b11-4bd2-99f7-fd667e55a985", "content-length": "2", "date": "Fri, 07 Mar 2025 22:45:53 GMT"} })
7162025-03-07T22:45:54.713ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
7172025-03-07T22:45:54.713ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
7182025-03-07T22:45:54.749ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
7192025-03-07T22:45:54.749ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "39914f82-032f-4609-bdb7-3eefab87d3c6", "date": "Fri, 07 Mar 2025 22:45:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
7202025-03-07T22:45:54.749ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
7212025-03-07T22:45:54.751ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "72ca85a3-ebd7-4a30-8640-07ba081d9a6a", "date": "Fri, 07 Mar 2025 22:45:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
7222025-03-07T22:45:54.751ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
7232025-03-07T22:45:54.753ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
7242025-03-07T22:45:54.753ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 339.721885ms iteration = 15 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
7252025-03-07T22:45:54.765ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 351.73131ms iteration = 25 status = {}
7262025-03-07T22:45:54.834ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
7272025-03-07T22:45:54.834ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "7248dde1-2ac4-4773-b642-0c639f255afe", "date": "Fri, 07 Mar 2025 22:45:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
7282025-03-07T22:45:54.834ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
7292025-03-07T22:45:54.836ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "f7113e8c-0ce6-4c41-8169-509fe1ed1fc6", "date": "Fri, 07 Mar 2025 22:45:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
7302025-03-07T22:45:54.836ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
7312025-03-07T22:45:54.838ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "87782402-b059-48a5-a0df-e5a4d0a47a35", "date": "Fri, 07 Mar 2025 22:45:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
7322025-03-07T22:45:54.838ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
7332025-03-07T22:45:54.839ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
7342025-03-07T22:45:54.839ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 426.385437ms iteration = 15 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
7352025-03-07T22:45:54.864ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
7362025-03-07T22:45:54.865ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7dc41f9-759d-4c25-adb2-a6dfd08f75bf", "content-length": "691", "date": "Fri, 07 Mar 2025 22:45:54 GMT"} })
7372025-03-07T22:45:54.865ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
7382025-03-07T22:45:54.865ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
7392025-03-07T22:45:54.865ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
7402025-03-07T22:45:54.867ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 453.886002ms iteration = 25 status = {}
7412025-03-07T22:45:54.870ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
7422025-03-07T22:45:54.870ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
7432025-03-07T22:45:54.870ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "cdf64570-bf2f-4e3e-bb6c-363012e0bc2c", "content-length": "372", "date": "Fri, 07 Mar 2025 22:45:54 GMT"} })
7442025-03-07T22:45:54.870ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
7452025-03-07T22:45:54.870ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
7462025-03-07T22:45:54.871ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 458.499996ms iteration = 25 status = null
7472025-03-07T22:45:54.909ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
7482025-03-07T22:45:54.909ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
7492025-03-07T22:45:54.909ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "9c92ee8f-6c06-40bd-a0b5-f69a5ea3b26f", "date": "Fri, 07 Mar 2025 22:45:54 GMT"} })
7502025-03-07T22:45:54.909ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
7512025-03-07T22:45:54.910ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
7522025-03-07T22:45:54.912ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
7532025-03-07T22:45:54.912ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
7542025-03-07T22:45:54.915ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
7552025-03-07T22:45:54.916ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
7562025-03-07T22:45:54.919ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
7572025-03-07T22:45:54.919ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
7582025-03-07T22:45:54.919ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "801236f7-22c0-41fa-afa2-dc23fd20931d", "date": "Fri, 07 Mar 2025 22:45:54 GMT"} })
7592025-03-07T22:45:54.921ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 508.356486ms iteration = 25 status = {}
7602025-03-07T22:45:54.937ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
7612025-03-07T22:45:54.937ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
7622025-03-07T22:45:54.939ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 2619304f-a0b9-451a-bbfb-b14af6353df6, rack_id: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(8820d7be-fc72-45ff-bc2e-17c4bb9613af) } rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
7632025-03-07T22:45:54.939ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
7642025-03-07T22:45:54.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
7652025-03-07T22:45:54.948ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
7662025-03-07T22:45:54.949ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "30152d40-d017-497e-8aa8-ac2dfb1a1847", "content-length": "134", "date": "Fri, 07 Mar 2025 22:45:54 GMT"} })
7672025-03-07T22:45:54.949ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
7682025-03-07T22:45:54.949ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 8820d7be-fc72-45ff-bc2e-17c4bb9613af
7692025-03-07T22:45:54.949ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
7702025-03-07T22:45:54.949ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
7712025-03-07T22:45:54.949ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
7722025-03-07T22:45:54.949ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
7732025-03-07T22:45:54.949ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
7742025-03-07T22:45:54.949ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
7752025-03-07T22:45:54.949ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
7762025-03-07T22:45:54.949ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
7772025-03-07T22:45:54.949ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
7782025-03-07T22:45:54.949ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
7792025-03-07T22:45:54.950ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
7802025-03-07T22:45:54.952ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
7812025-03-07T22:45:54.952ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "55186ee7-bced-497a-8e30-7cf34b461a81", "content-length": "2", "date": "Fri, 07 Mar 2025 22:45:54 GMT"} })
7822025-03-07T22:45:54.952ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
7832025-03-07T22:45:54.952ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "475d849d-4e96-4fa7-8bdc-f6a19bf7e518", "content-length": "52", "date": "Fri, 07 Mar 2025 22:45:54 GMT"} })
7842025-03-07T22:45:54.952ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
7852025-03-07T22:45:54.952ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
7862025-03-07T22:45:55.054ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "f6bcaf83-5770-4fd4-9c97-8a048d68045c", "date": "Fri, 07 Mar 2025 22:45:54 GMT"} })
7872025-03-07T22:45:55.054ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch = Switch0
7882025-03-07T22:45:55.054ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
7892025-03-07T22:45:55.055ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4fcb7cc-c7c5-455b-b926-4743666f3a6e", "content-length": "682", "date": "Fri, 07 Mar 2025 22:45:54 GMT"} })
7902025-03-07T22:45:55.055ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
7912025-03-07T22:45:55.055ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
7922025-03-07T22:45:55.059ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
7932025-03-07T22:45:55.060ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
7942025-03-07T22:45:55.063ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
7952025-03-07T22:45:55.063ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
7962025-03-07T22:45:55.065ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
7972025-03-07T22:45:55.066ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
7982025-03-07T22:45:55.074ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 660.555558ms iteration = 25 status = {}
7992025-03-07T22:46:04.413ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 74 reason = Timeout
8002025-03-07T22:46:04.413ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 74 reason = Timeout
8012025-03-07T22:46:04.413ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8022025-03-07T22:46:04.413ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
8032025-03-07T22:46:04.414ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
8042025-03-07T22:46:04.414ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8052025-03-07T22:46:04.416ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8062025-03-07T22:46:04.419ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
8072025-03-07T22:46:04.419ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8082025-03-07T22:46:04.419ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8092025-03-07T22:46:04.421ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8102025-03-07T22:46:04.421ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8112025-03-07T22:46:04.423ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8122025-03-07T22:46:04.426ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
8132025-03-07T22:46:04.426ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8142025-03-07T22:46:04.426ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8152025-03-07T22:46:04.436ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
8162025-03-07T22:46:04.436ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.509866ms iteration = 74 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
8172025-03-07T22:46:04.444ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
8182025-03-07T22:46:04.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
8192025-03-07T22:46:04.474ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 61.211088ms iteration = 74 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
8202025-03-07T22:46:04.680ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 375 local_addr = [fd00:1122:3344:101::c]:41480 method = GET remote_addr = [fd00:1122:3344:101::d]:63523 req_id = 3c2b785c-4558-451c-a39d-c4874593bc5d response_code = 200 uri = /c0f69897-7f92-43db-ad47-96c3c4721128
8212025-03-07T22:46:14.414ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 75 reason = Timeout
8222025-03-07T22:46:14.414ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 75 reason = Timeout
8232025-03-07T22:46:14.414ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
8242025-03-07T22:46:14.414ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8252025-03-07T22:46:14.414ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8262025-03-07T22:46:14.414ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
8272025-03-07T22:46:14.416ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8282025-03-07T22:46:14.418ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
8292025-03-07T22:46:14.418ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8302025-03-07T22:46:14.419ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8312025-03-07T22:46:14.421ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8322025-03-07T22:46:14.421ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8332025-03-07T22:46:14.422ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8342025-03-07T22:46:14.425ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
8352025-03-07T22:46:14.425ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8362025-03-07T22:46:14.425ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8372025-03-07T22:46:14.438ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
8382025-03-07T22:46:14.438ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 24.667967ms iteration = 75 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
8392025-03-07T22:46:14.444ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
8402025-03-07T22:46:14.444ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
8412025-03-07T22:46:14.472ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 58.099428ms iteration = 75 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
8422025-03-07T22:46:14.680ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 568 local_addr = [fd00:1122:3344:101::c]:41480 method = GET remote_addr = [fd00:1122:3344:101::d]:63523 req_id = a810c010-dcc9-419a-88cd-c6be2623d6e6 response_code = 200 uri = /c0f69897-7f92-43db-ad47-96c3c4721128
8432025-03-07T22:46:24.414ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 76 reason = Timeout
8442025-03-07T22:46:24.414ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8452025-03-07T22:46:24.414ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 26 reason = Timeout
8462025-03-07T22:46:24.414ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 26 reason = Timeout
8472025-03-07T22:46:24.414ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
8482025-03-07T22:46:24.414ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
8492025-03-07T22:46:24.414ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 28 reason = Timeout
8502025-03-07T22:46:24.414ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 26 reason = Timeout
8512025-03-07T22:46:24.414ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
8522025-03-07T22:46:24.414ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
8532025-03-07T22:46:24.414ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 26 reason = Timeout
8542025-03-07T22:46:24.414ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 26 reason = Timeout
8552025-03-07T22:46:24.414ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
8562025-03-07T22:46:24.414ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 26 reason = Timeout
8572025-03-07T22:46:24.414ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
8582025-03-07T22:46:24.414ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 76 reason = Timeout
8592025-03-07T22:46:24.414ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 26 reason = Timeout
8602025-03-07T22:46:24.414ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
8612025-03-07T22:46:24.414ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 26 reason = Timeout
8622025-03-07T22:46:24.415ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
8632025-03-07T22:46:24.415ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
8642025-03-07T22:46:24.415ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
8652025-03-07T22:46:24.415ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 26 reason = Timeout
8662025-03-07T22:46:24.415ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
8672025-03-07T22:46:24.415ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 26 reason = Timeout
8682025-03-07T22:46:24.415ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
8692025-03-07T22:46:24.415ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
8702025-03-07T22:46:24.415ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 26 reason = Timeout
8712025-03-07T22:46:24.415ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
8722025-03-07T22:46:24.415ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 26 reason = Timeout
8732025-03-07T22:46:24.415ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
8742025-03-07T22:46:24.415ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 26 reason = Timeout
8752025-03-07T22:46:24.415ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
8762025-03-07T22:46:24.415ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 26 reason = Timeout
8772025-03-07T22:46:24.415ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
8782025-03-07T22:46:24.415ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 26 reason = Timeout
8792025-03-07T22:46:24.415ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
8802025-03-07T22:46:24.415ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 26 reason = Timeout
8812025-03-07T22:46:24.415ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
8822025-03-07T22:46:24.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8832025-03-07T22:46:24.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
8842025-03-07T22:46:24.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
8852025-03-07T22:46:24.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
8862025-03-07T22:46:24.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
8872025-03-07T22:46:24.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
8882025-03-07T22:46:24.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
8892025-03-07T22:46:24.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
8902025-03-07T22:46:24.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
8912025-03-07T22:46:24.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
8922025-03-07T22:46:24.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
8932025-03-07T22:46:24.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
8942025-03-07T22:46:24.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
8952025-03-07T22:46:24.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
8962025-03-07T22:46:24.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
8972025-03-07T22:46:24.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
8982025-03-07T22:46:24.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
8992025-03-07T22:46:24.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
9002025-03-07T22:46:24.417ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9012025-03-07T22:46:24.418ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9022025-03-07T22:46:24.418ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.971778ms iteration = 26 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
9032025-03-07T22:46:24.418ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1542, tv_nsec: 764329324 } })
9042025-03-07T22:46:24.419ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
9052025-03-07T22:46:24.420ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
9062025-03-07T22:46:24.420ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9072025-03-07T22:46:24.420ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9082025-03-07T22:46:24.420ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
9092025-03-07T22:46:24.420ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
9102025-03-07T22:46:24.421ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
9112025-03-07T22:46:24.422ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9122025-03-07T22:46:24.423ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9132025-03-07T22:46:24.423ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1542, tv_nsec: 768916438 } })
9142025-03-07T22:46:24.423ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9152025-03-07T22:46:24.423ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
9162025-03-07T22:46:24.424ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
9172025-03-07T22:46:24.424ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
9182025-03-07T22:46:24.425ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
9192025-03-07T22:46:24.425ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
9202025-03-07T22:46:24.425ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
9212025-03-07T22:46:24.425ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
9222025-03-07T22:46:24.426ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
9232025-03-07T22:46:24.427ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
9242025-03-07T22:46:24.428ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
9252025-03-07T22:46:24.428ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9262025-03-07T22:46:24.428ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9272025-03-07T22:46:24.432ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9282025-03-07T22:46:24.432ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
9292025-03-07T22:46:24.432ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9302025-03-07T22:46:24.433ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9312025-03-07T22:46:24.433ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
9322025-03-07T22:46:24.433ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
9332025-03-07T22:46:24.433ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
9342025-03-07T22:46:24.433ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
9352025-03-07T22:46:24.434ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
9362025-03-07T22:46:24.434ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9372025-03-07T22:46:24.434ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9382025-03-07T22:46:24.435ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
9392025-03-07T22:46:24.435ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
9402025-03-07T22:46:24.435ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
9412025-03-07T22:46:24.435ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9422025-03-07T22:46:24.438ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
9432025-03-07T22:46:24.438ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
9442025-03-07T22:46:24.438ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
9452025-03-07T22:46:24.461ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9462025-03-07T22:46:24.462ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 47.603027ms iteration = 26 status = {"errors":[],"finish_invoked_ok":[]}
9472025-03-07T22:46:24.463ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
9482025-03-07T22:46:24.463ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
9492025-03-07T22:46:24.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
9502025-03-07T22:46:24.468ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1542, tv_nsec: 813587327 } })
9512025-03-07T22:46:24.468ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
9522025-03-07T22:46:24.469ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9532025-03-07T22:46:24.470ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
9542025-03-07T22:46:24.470ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 55.801647ms iteration = 26 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
9552025-03-07T22:46:24.471ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
9562025-03-07T22:46:24.471ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
9572025-03-07T22:46:24.472ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
9582025-03-07T22:46:24.472ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
9592025-03-07T22:46:24.472ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
9602025-03-07T22:46:24.474ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 59.693913ms iteration = 26 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
9612025-03-07T22:46:24.474ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9622025-03-07T22:46:24.474ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9632025-03-07T22:46:24.475ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
9642025-03-07T22:46:24.475ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
9652025-03-07T22:46:24.476ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
9662025-03-07T22:46:24.477ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9672025-03-07T22:46:24.477ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
9682025-03-07T22:46:24.477ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9692025-03-07T22:46:24.477ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9702025-03-07T22:46:24.479ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 64.050516ms iteration = 26 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
9712025-03-07T22:46:24.479ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9722025-03-07T22:46:24.480ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 65.236247ms iteration = 26 status = {"errors":[],"garbage_collect_requested":[]}
9732025-03-07T22:46:24.480ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
9742025-03-07T22:46:24.480ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9752025-03-07T22:46:24.480ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9762025-03-07T22:46:24.482ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 73b218dd-7dbc-4345-acb1-fe69ced4a7f7, lookup_type: ById(73b218dd-7dbc-4345-acb1-fe69ced4a7f7) } result = Ok(())
9772025-03-07T22:46:24.482ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9782025-03-07T22:46:24.482ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9792025-03-07T22:46:24.483ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
9802025-03-07T22:46:24.483ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
9812025-03-07T22:46:24.483ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
9822025-03-07T22:46:24.483ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 68.67218ms iteration = 76 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
9832025-03-07T22:46:24.483ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
9842025-03-07T22:46:24.483ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 69.001113ms iteration = 26 status = 0
9852025-03-07T22:46:24.484ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
9862025-03-07T22:46:24.484ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
9872025-03-07T22:46:24.484ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d957d19-56d5-410c-a059-be4edffd43ae", "content-length": "26", "date": "Fri, 07 Mar 2025 22:46:24 GMT"} })
9882025-03-07T22:46:24.484ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
9892025-03-07T22:46:24.486ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
9902025-03-07T22:46:24.487ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 72.943961ms iteration = 26 status = {"errors":[],"requests_created_ok":[]}
9912025-03-07T22:46:24.493ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
9922025-03-07T22:46:24.493ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 78.400704ms iteration = 76 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
9932025-03-07T22:46:24.508ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
9942025-03-07T22:46:24.508ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
9952025-03-07T22:46:24.508ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e07adc0-8f10-45fc-88cb-022c0202579f", "content-length": "26", "date": "Fri, 07 Mar 2025 22:46:24 GMT"} })
9962025-03-07T22:46:24.508ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
9972025-03-07T22:46:24.510ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
9982025-03-07T22:46:24.513ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
9992025-03-07T22:46:24.513ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
10002025-03-07T22:46:24.515ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
10012025-03-07T22:46:24.515ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
10022025-03-07T22:46:24.518ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
10032025-03-07T22:46:24.518ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
10042025-03-07T22:46:24.519ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10052025-03-07T22:46:24.522ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
10062025-03-07T22:46:24.522ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
10072025-03-07T22:46:24.522ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
10082025-03-07T22:46:24.522ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
10092025-03-07T22:46:24.522ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
10102025-03-07T22:46:24.522ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 107.817406ms iteration = 26 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
10112025-03-07T22:46:24.523ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
10122025-03-07T22:46:24.523ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 109.075238ms iteration = 26 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
10132025-03-07T22:46:24.544ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 128.97672ms iteration = 26 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
10142025-03-07T22:46:24.559ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec zone_address = fd00:1122:3344:101::2
10152025-03-07T22:46:24.559ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
10162025-03-07T22:46:24.559ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "c903fbc3-769a-48b3-94f5-6f1044f7f08a", "content-length": "26", "date": "Fri, 07 Mar 2025 22:46:24 GMT"} })
10172025-03-07T22:46:24.559ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
10182025-03-07T22:46:24.561ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
10192025-03-07T22:46:24.565ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
10202025-03-07T22:46:24.565ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
10212025-03-07T22:46:24.566ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10222025-03-07T22:46:24.569ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
10232025-03-07T22:46:24.569ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
10242025-03-07T22:46:24.569ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
10252025-03-07T22:46:24.571ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10262025-03-07T22:46:24.573ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
10272025-03-07T22:46:24.573ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
10282025-03-07T22:46:24.573ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
10292025-03-07T22:46:24.577ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 162.866225ms iteration = 28 status = {"physical_disks_added":0}
10302025-03-07T22:46:24.580ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 165.284819ms iteration = 26 status = {}
10312025-03-07T22:46:24.608ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
10322025-03-07T22:46:24.609ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "08fd6fe7-e792-45c9-b5dc-4615c64e1f13", "content-length": "2", "date": "Fri, 07 Mar 2025 22:46:23 GMT"} })
10332025-03-07T22:46:24.610ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
10342025-03-07T22:46:24.611ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
10352025-03-07T22:46:24.613ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 198.099034ms iteration = 26 status = null
10362025-03-07T22:46:24.669ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
10372025-03-07T22:46:24.669ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d7e87c5-22f9-4fee-87a2-e2d682329746", "content-length": "691", "date": "Fri, 07 Mar 2025 22:46:24 GMT"} })
10382025-03-07T22:46:24.669ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
10392025-03-07T22:46:24.669ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
10402025-03-07T22:46:24.669ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
10412025-03-07T22:46:24.670ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 255.697918ms iteration = 26 status = {}
10422025-03-07T22:46:24.680ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 306 local_addr = [fd00:1122:3344:101::c]:41480 method = GET remote_addr = [fd00:1122:3344:101::d]:63523 req_id = fd0b2cdb-25da-4841-b5c7-c47791022e84 response_code = 200 uri = /c0f69897-7f92-43db-ad47-96c3c4721128
10432025-03-07T22:46:24.689ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
10442025-03-07T22:46:24.689ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
10452025-03-07T22:46:24.689ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "5cb51f9b-b258-4d09-b95c-8709b0acae94", "content-length": "372", "date": "Fri, 07 Mar 2025 22:46:24 GMT"} })
10462025-03-07T22:46:24.689ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
10472025-03-07T22:46:24.689ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
10482025-03-07T22:46:24.730ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
10492025-03-07T22:46:24.731ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
10502025-03-07T22:46:24.731ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "dec957df-0e60-43cf-b73e-cfafb5fb571d", "date": "Fri, 07 Mar 2025 22:46:24 GMT"} })
10512025-03-07T22:46:24.731ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
10522025-03-07T22:46:24.731ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
10532025-03-07T22:46:24.733ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
10542025-03-07T22:46:24.734ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
10552025-03-07T22:46:24.737ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
10562025-03-07T22:46:24.737ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
10572025-03-07T22:46:24.740ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
10582025-03-07T22:46:24.740ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
10592025-03-07T22:46:24.740ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "3ba3898a-5a13-4d53-980a-6c45ba28fca4", "date": "Fri, 07 Mar 2025 22:46:24 GMT"} })
10602025-03-07T22:46:24.742ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 327.106823ms iteration = 26 status = {}
10612025-03-07T22:46:24.757ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
10622025-03-07T22:46:24.757ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
10632025-03-07T22:46:24.760ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 2619304f-a0b9-451a-bbfb-b14af6353df6, rack_id: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(8820d7be-fc72-45ff-bc2e-17c4bb9613af) } rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
10642025-03-07T22:46:24.760ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
10652025-03-07T22:46:24.760ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
10662025-03-07T22:46:24.769ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
10672025-03-07T22:46:24.770ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "39ada9c2-82af-462b-b1be-5360ca6d7d7c", "content-length": "134", "date": "Fri, 07 Mar 2025 22:46:24 GMT"} })
10682025-03-07T22:46:24.770ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
10692025-03-07T22:46:24.770ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 8820d7be-fc72-45ff-bc2e-17c4bb9613af
10702025-03-07T22:46:24.770ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
10712025-03-07T22:46:24.770ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
10722025-03-07T22:46:24.770ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
10732025-03-07T22:46:24.770ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
10742025-03-07T22:46:24.770ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
10752025-03-07T22:46:24.770ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
10762025-03-07T22:46:24.770ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
10772025-03-07T22:46:24.770ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
10782025-03-07T22:46:24.770ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
10792025-03-07T22:46:24.770ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
10802025-03-07T22:46:24.771ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
10812025-03-07T22:46:24.772ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
10822025-03-07T22:46:24.772ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f9edfcb-91ce-40db-960c-91dc67f4e7b2", "content-length": "2", "date": "Fri, 07 Mar 2025 22:46:24 GMT"} })
10832025-03-07T22:46:24.772ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
10842025-03-07T22:46:24.772ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c015b2b-a817-4bb1-9264-cceaa4029884", "content-length": "52", "date": "Fri, 07 Mar 2025 22:46:24 GMT"} })
10852025-03-07T22:46:24.772ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
10862025-03-07T22:46:24.772ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
10872025-03-07T22:46:24.862ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "ad5c19a2-4dd8-4040-8037-af307ace0b29", "date": "Fri, 07 Mar 2025 22:46:24 GMT"} })
10882025-03-07T22:46:24.862ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch = Switch0
10892025-03-07T22:46:24.862ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
10902025-03-07T22:46:24.862ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6357c71-fbec-47a5-a680-f9309f911e8a", "content-length": "682", "date": "Fri, 07 Mar 2025 22:46:24 GMT"} })
10912025-03-07T22:46:24.862ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
10922025-03-07T22:46:24.863ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
10932025-03-07T22:46:24.866ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
10942025-03-07T22:46:24.866ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
10952025-03-07T22:46:24.869ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
10962025-03-07T22:46:24.870ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
10972025-03-07T22:46:24.872ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
10982025-03-07T22:46:24.872ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
10992025-03-07T22:46:24.879ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 464.384363ms iteration = 26 status = {}
11002025-03-07T22:46:34.413ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 77 reason = Timeout
11012025-03-07T22:46:34.413ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 77 reason = Timeout
11022025-03-07T22:46:34.414ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
11032025-03-07T22:46:34.414ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
11042025-03-07T22:46:34.414ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
11052025-03-07T22:46:34.414ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
11062025-03-07T22:46:34.415ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11072025-03-07T22:46:34.416ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
11082025-03-07T22:46:34.416ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
11092025-03-07T22:46:34.418ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.499133ms iteration = 77 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
11102025-03-07T22:46:34.418ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
11112025-03-07T22:46:34.418ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
11122025-03-07T22:46:34.418ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
11132025-03-07T22:46:34.420ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
11142025-03-07T22:46:34.420ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
11152025-03-07T22:46:34.422ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11162025-03-07T22:46:34.424ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
11172025-03-07T22:46:34.424ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
11182025-03-07T22:46:34.424ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
11192025-03-07T22:46:34.434ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
11202025-03-07T22:46:34.434ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.620178ms iteration = 77 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
11212025-03-07T22:46:34.680ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 310 local_addr = [fd00:1122:3344:101::c]:41480 method = GET remote_addr = [fd00:1122:3344:101::d]:63523 req_id = a6176f13-ece4-4add-8a65-5f84b140f9be response_code = 200 uri = /c0f69897-7f92-43db-ad47-96c3c4721128
11222025-03-07T22:46:44.414ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 78 reason = Timeout
11232025-03-07T22:46:44.414ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 78 reason = Timeout
11242025-03-07T22:46:44.414ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
11252025-03-07T22:46:44.414ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
11262025-03-07T22:46:44.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
11272025-03-07T22:46:44.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
11282025-03-07T22:46:44.416ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
11292025-03-07T22:46:44.417ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
11302025-03-07T22:46:44.417ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11312025-03-07T22:46:44.420ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
11322025-03-07T22:46:44.420ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
11332025-03-07T22:46:44.420ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
11342025-03-07T22:46:44.422ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
11352025-03-07T22:46:44.422ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
11362025-03-07T22:46:44.423ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11372025-03-07T22:46:44.426ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
11382025-03-07T22:46:44.426ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
11392025-03-07T22:46:44.426ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
11402025-03-07T22:46:44.435ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
11412025-03-07T22:46:44.435ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.068223ms iteration = 78 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
11422025-03-07T22:46:44.445ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 30.971418ms iteration = 78 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
11432025-03-07T22:46:44.681ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 473 local_addr = [fd00:1122:3344:101::c]:41480 method = GET remote_addr = [fd00:1122:3344:101::d]:63523 req_id = e3bc00bc-6ff2-40cb-b6b9-88eaf229e350 response_code = 200 uri = /c0f69897-7f92-43db-ad47-96c3c4721128
11442025-03-07T22:46:54.414ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 29 reason = Timeout
11452025-03-07T22:46:54.414ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 15 reason = Timeout
11462025-03-07T22:46:54.414ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f
11472025-03-07T22:46:54.414ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
11482025-03-07T22:46:54.414ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 13.98\u{b5}s iteration = 15 status = {"enabled":false,"target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f"}
11492025-03-07T22:46:54.414ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 16 reason = Timeout
11502025-03-07T22:46:54.414ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 14 reason = Timeout
11512025-03-07T22:46:54.414ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 27 reason = Timeout
11522025-03-07T22:46:54.414ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
11532025-03-07T22:46:54.414ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11542025-03-07T22:46:54.414ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 79 reason = Timeout
11552025-03-07T22:46:54.414ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
11562025-03-07T22:46:54.414ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 27 reason = Timeout
11572025-03-07T22:46:54.414ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 27 reason = Timeout
11582025-03-07T22:46:54.414ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
11592025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
11602025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 14 reason = Timeout
11612025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
11622025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
11632025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 79 reason = Timeout
11642025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 16 reason = Timeout
11652025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
11662025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 27 reason = Timeout
11672025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 14 reason = Timeout
11682025-03-07T22:46:54.415ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
11692025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
11702025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
11712025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 27 reason = Timeout
11722025-03-07T22:46:54.415ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
11732025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
11742025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 27 reason = Timeout
11752025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
11762025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 14 reason = Timeout
11772025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 27 reason = Timeout
11782025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
11792025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
11802025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
11812025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 27 reason = Timeout
11822025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
11832025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 14 reason = Timeout
11842025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
11852025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
11862025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 27 reason = Timeout
11872025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
11882025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 14 reason = Timeout
11892025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 14 reason = Timeout
11902025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 14 reason = Timeout
11912025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
11922025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 14 reason = Timeout
11932025-03-07T22:46:54.415ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-07 22:36:54.415131323 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
11942025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
11952025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 14 reason = Timeout
11962025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
11972025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 27 reason = Timeout
11982025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11992025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 27 reason = Timeout
12002025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 27 reason = Timeout
12012025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
12022025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
12032025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 27 reason = Timeout
12042025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
12052025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 27 reason = Timeout
12062025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
12072025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
12082025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 27 reason = Timeout
12092025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
12102025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 27 reason = Timeout
12112025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
12122025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
12132025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
12142025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
12152025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
12162025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
12172025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
12182025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
12192025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
12202025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12212025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
12222025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
12232025-03-07T22:46:54.415ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("732b0e55-5ccf-4068-b771-1299f872f618.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("fbf58a59-609b-4fb3-b792-2c09f3f7b776.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("fbf58a59-609b-4fb3-b792-2c09f3f7b776.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 1572, tv_nsec: 760645568 } })
12242025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
12252025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
12262025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
12272025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
12282025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
12292025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
12302025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
12312025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
12322025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
12332025-03-07T22:46:54.415ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("114e494f-6e94-4e0c-80db-5823b226f1a3.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("a9773e0b-15d4-4efa-b977-9f3598685cdd.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("d2f04f38-7ac2-4327-a087-c2a47865841a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("d2f04f38-7ac2-4327-a087-c2a47865841a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 1572, tv_nsec: 760761389 } })
12342025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
12352025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
12362025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
12372025-03-07T22:46:54.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
12382025-03-07T22:46:54.416ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
12392025-03-07T22:46:54.416ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 997.149\u{b5}s iteration = 14 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
12402025-03-07T22:46:54.416ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
12412025-03-07T22:46:54.416ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.172531ms iteration = 14 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353"]}
12422025-03-07T22:46:54.418ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1572, tv_nsec: 763275633 } })
12432025-03-07T22:46:54.418ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
12442025-03-07T22:46:54.418ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12452025-03-07T22:46:54.418ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.651725ms iteration = 27 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
12462025-03-07T22:46:54.419ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12472025-03-07T22:46:54.419ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
12482025-03-07T22:46:54.419ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
12492025-03-07T22:46:54.420ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 5.452732ms iteration = 14 status = {"errors":[],"found_port_ok":[]}
12502025-03-07T22:46:54.420ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12512025-03-07T22:46:54.422ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
12522025-03-07T22:46:54.422ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
12532025-03-07T22:46:54.422ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
12542025-03-07T22:46:54.422ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
12552025-03-07T22:46:54.422ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
12562025-03-07T22:46:54.422ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
12572025-03-07T22:46:54.422ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
12582025-03-07T22:46:54.422ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
12592025-03-07T22:46:54.423ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
12602025-03-07T22:46:54.423ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
12612025-03-07T22:46:54.423ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
12622025-03-07T22:46:54.424ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12632025-03-07T22:46:54.425ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12642025-03-07T22:46:54.425ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12652025-03-07T22:46:54.425ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
12662025-03-07T22:46:54.425ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
12672025-03-07T22:46:54.425ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
12682025-03-07T22:46:54.425ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
12692025-03-07T22:46:54.426ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12702025-03-07T22:46:54.427ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
12712025-03-07T22:46:54.427ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
12722025-03-07T22:46:54.427ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
12732025-03-07T22:46:54.428ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
12742025-03-07T22:46:54.428ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
12752025-03-07T22:46:54.428ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
12762025-03-07T22:46:54.428ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
12772025-03-07T22:46:54.428ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
12782025-03-07T22:46:54.429ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
12792025-03-07T22:46:54.430ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
12802025-03-07T22:46:54.430ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12812025-03-07T22:46:54.430ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12822025-03-07T22:46:54.430ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12832025-03-07T22:46:54.430ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
12842025-03-07T22:46:54.430ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1572, tv_nsec: 775825864 } })
12852025-03-07T22:46:54.431ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
12862025-03-07T22:46:54.431ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
12872025-03-07T22:46:54.432ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
12882025-03-07T22:46:54.432ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12892025-03-07T22:46:54.432ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
12902025-03-07T22:46:54.432ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
12912025-03-07T22:46:54.432ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
12922025-03-07T22:46:54.432ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 17.807831ms iteration = 14 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
12932025-03-07T22:46:54.432ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
12942025-03-07T22:46:54.433ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
12952025-03-07T22:46:54.433ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
12962025-03-07T22:46:54.435ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-07 22:36:54.415131323 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
12972025-03-07T22:46:54.435ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 20.180634ms iteration = 14 status = {"expiration":"2025-03-07T22:36:54.415131323Z","pruned":[]}
12982025-03-07T22:46:54.436ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
12992025-03-07T22:46:54.436ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
13002025-03-07T22:46:54.436ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
13012025-03-07T22:46:54.438ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 24.041111ms iteration = 79 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
13022025-03-07T22:46:54.440ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13032025-03-07T22:46:54.440ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
13042025-03-07T22:46:54.440ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1572, tv_nsec: 785620758 } })
13052025-03-07T22:46:54.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
13062025-03-07T22:46:54.440ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
13072025-03-07T22:46:54.441ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13082025-03-07T22:46:54.443ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
13092025-03-07T22:46:54.443ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
13102025-03-07T22:46:54.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
13112025-03-07T22:46:54.444ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
13122025-03-07T22:46:54.446ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
13132025-03-07T22:46:54.447ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
13142025-03-07T22:46:54.449ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
13152025-03-07T22:46:54.449ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
13162025-03-07T22:46:54.449ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
13172025-03-07T22:46:54.449ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
13182025-03-07T22:46:54.449ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13192025-03-07T22:46:54.451ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13202025-03-07T22:46:54.453ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
13212025-03-07T22:46:54.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
13222025-03-07T22:46:54.454ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
13232025-03-07T22:46:54.454ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
13242025-03-07T22:46:54.454ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
13252025-03-07T22:46:54.454ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
13262025-03-07T22:46:54.454ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
13272025-03-07T22:46:54.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
13282025-03-07T22:46:54.457ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13292025-03-07T22:46:54.458ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
13302025-03-07T22:46:54.458ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 43.453658ms iteration = 27 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
13312025-03-07T22:46:54.461ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
13322025-03-07T22:46:54.461ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
13332025-03-07T22:46:54.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
13342025-03-07T22:46:54.465ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 51.087411ms iteration = 29 status = {"physical_disks_added":0}
13352025-03-07T22:46:54.469ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
13362025-03-07T22:46:54.470ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
13372025-03-07T22:46:54.472ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
13382025-03-07T22:46:54.472ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
13392025-03-07T22:46:54.474ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
13402025-03-07T22:46:54.474ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
13412025-03-07T22:46:54.474ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
13422025-03-07T22:46:54.474ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 60.056237ms iteration = 79 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
13432025-03-07T22:46:54.476ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 61.161867ms iteration = 27 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
13442025-03-07T22:46:54.478ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 63.092726ms iteration = 27 status = {"errors":[],"finish_invoked_ok":[]}
13452025-03-07T22:46:54.480ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
13462025-03-07T22:46:54.480ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
13472025-03-07T22:46:54.480ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
13482025-03-07T22:46:54.480ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
13492025-03-07T22:46:54.486ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 70.969833ms iteration = 27 status = {"errors":[],"garbage_collect_requested":[]}
13502025-03-07T22:46:54.487ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
13512025-03-07T22:46:54.487ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
13522025-03-07T22:46:54.488ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
13532025-03-07T22:46:54.488ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 73.744859ms iteration = 27 status = {"errors":[],"requests_created_ok":[]}
13542025-03-07T22:46:54.489ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
13552025-03-07T22:46:54.490ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
13562025-03-07T22:46:54.490ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
13572025-03-07T22:46:54.492ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13582025-03-07T22:46:54.492ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13592025-03-07T22:46:54.495ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 73b218dd-7dbc-4345-acb1-fe69ced4a7f7, lookup_type: ById(73b218dd-7dbc-4345-acb1-fe69ced4a7f7) } result = Ok(())
13602025-03-07T22:46:54.495ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
13612025-03-07T22:46:54.495ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
13622025-03-07T22:46:54.496ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
13632025-03-07T22:46:54.497ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
13642025-03-07T22:46:54.497ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
13652025-03-07T22:46:54.497ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
13662025-03-07T22:46:54.497ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
13672025-03-07T22:46:54.497ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
13682025-03-07T22:46:54.497ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
13692025-03-07T22:46:54.497ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 82.984417ms iteration = 27 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
13702025-03-07T22:46:54.499ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13712025-03-07T22:46:54.502ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
13722025-03-07T22:46:54.502ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
13732025-03-07T22:46:54.502ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
13742025-03-07T22:46:54.504ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
13752025-03-07T22:46:54.504ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = a49f17b2-1834-4c40-bc6e-ff0588e4b3ef dns_zone_name = oxide.test found_so_far = 0 version = 2
13762025-03-07T22:46:54.504ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
13772025-03-07T22:46:54.505ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
13782025-03-07T22:46:54.506ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13792025-03-07T22:46:54.507ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 92.244176ms iteration = 27 status = 0
13802025-03-07T22:46:54.508ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 93.761941ms iteration = 27 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
13812025-03-07T22:46:54.510ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
13822025-03-07T22:46:54.510ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
13832025-03-07T22:46:54.510ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
13842025-03-07T22:46:54.514ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
13852025-03-07T22:46:54.514ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
13862025-03-07T22:46:54.514ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 99.88554ms iteration = 14 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
13872025-03-07T22:46:54.518ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = a49f17b2-1834-4c40-bc6e-ff0588e4b3ef dns_zone_name = oxide.test found_so_far = 1 version = 2
13882025-03-07T22:46:54.518ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
13892025-03-07T22:46:54.518ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-07 22:34:17.797135 UTC dns_group = external generation = 2
13902025-03-07T22:46:54.518ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 103.948369ms iteration = 14 status = {"generation":2}
13912025-03-07T22:46:54.519ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13922025-03-07T22:46:54.520ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
13932025-03-07T22:46:54.521ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
13942025-03-07T22:46:54.522ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
13952025-03-07T22:46:54.522ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
13962025-03-07T22:46:54.522ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
13972025-03-07T22:46:54.524ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
13982025-03-07T22:46:54.524ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
13992025-03-07T22:46:54.524ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 109.628594ms iteration = 27 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
14002025-03-07T22:46:54.541ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
14012025-03-07T22:46:54.541ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
14022025-03-07T22:46:54.541ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
14032025-03-07T22:46:54.541ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
14042025-03-07T22:46:54.541ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
14052025-03-07T22:46:54.542ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 127.373004ms iteration = 27 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
14062025-03-07T22:46:54.543ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14072025-03-07T22:46:54.545ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
14082025-03-07T22:46:54.546ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
14092025-03-07T22:46:54.546ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
14102025-03-07T22:46:54.561ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
14112025-03-07T22:46:54.562ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
14122025-03-07T22:46:54.563ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14132025-03-07T22:46:54.565ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
14142025-03-07T22:46:54.565ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
14152025-03-07T22:46:54.565ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
14162025-03-07T22:46:54.576ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
14172025-03-07T22:46:54.577ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = fdc30f7a-36a7-4893-91d6-3456da7362c0 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
14182025-03-07T22:46:54.577ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
14192025-03-07T22:46:54.577ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
14202025-03-07T22:46:54.578ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14212025-03-07T22:46:54.579ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
14222025-03-07T22:46:54.579ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
14232025-03-07T22:46:54.580ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14242025-03-07T22:46:54.583ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
14252025-03-07T22:46:54.583ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
14262025-03-07T22:46:54.583ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
14272025-03-07T22:46:54.584ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
14282025-03-07T22:46:54.584ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
14292025-03-07T22:46:54.584ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
14302025-03-07T22:46:54.585ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
14312025-03-07T22:46:54.585ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
14322025-03-07T22:46:54.585ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec11470f-7f04-4cdd-b70c-ce3e0b40a325", "content-length": "26", "date": "Fri, 07 Mar 2025 22:46:54 GMT"} })
14332025-03-07T22:46:54.585ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
14342025-03-07T22:46:54.585ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = fdc30f7a-36a7-4893-91d6-3456da7362c0 dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
14352025-03-07T22:46:54.585ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
14362025-03-07T22:46:54.585ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-07 22:34:17.208516 UTC dns_group = internal generation = 1
14372025-03-07T22:46:54.585ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 170.56234ms iteration = 14 status = {"generation":1}
14382025-03-07T22:46:54.585ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14392025-03-07T22:46:54.586ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
14402025-03-07T22:46:54.586ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
14412025-03-07T22:46:54.587ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
14422025-03-07T22:46:54.587ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
14432025-03-07T22:46:54.587ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
14442025-03-07T22:46:54.611ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-07T22:33:52.171499Z, time_modified: 2025-03-07T22:33:52.171499Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, db_silo: Silo { identity: SiloIdentity { id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-07T22:34:17.695388Z, time_modified: 2025-03-07T22:34:17.695388Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, db_silo: Silo { identity: SiloIdentity { id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-07T22:34:17.695388Z, time_modified: 2025-03-07T22:34:17.695388Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1" }] }) }
14452025-03-07T22:46:54.612ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 196.825731ms iteration = 14 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"ebb5815c-a32b-49c0-a621-3a7cd3eb3452","tls_certs":["f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1"]}},"default_endpoint":{"silo_id":"ebb5815c-a32b-49c0-a621-3a7cd3eb3452","tls_certs":["f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
14462025-03-07T22:46:54.616ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
14472025-03-07T22:46:54.616ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 201.152553ms iteration = 14 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
14482025-03-07T22:46:54.654ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
14492025-03-07T22:46:54.668ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
14502025-03-07T22:46:54.668ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
14512025-03-07T22:46:54.670ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
14522025-03-07T22:46:54.670ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
14532025-03-07T22:46:54.670ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e1cc1dc-3cbe-49bb-88f1-8a0f35cfa253", "content-length": "26", "date": "Fri, 07 Mar 2025 22:46:54 GMT"} })
14542025-03-07T22:46:54.671ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
14552025-03-07T22:46:54.671ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "24286c44-42fb-4a1c-b519-3709343af7e8", "content-length": "372", "date": "Fri, 07 Mar 2025 22:46:54 GMT"} })
14562025-03-07T22:46:54.671ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14572025-03-07T22:46:54.671ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14582025-03-07T22:46:54.672ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
14592025-03-07T22:46:54.681ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 329 local_addr = [fd00:1122:3344:101::c]:41480 method = GET remote_addr = [fd00:1122:3344:101::d]:63523 req_id = 22945d76-dd39-47dd-90fe-1d7eff3a5792 response_code = 200 uri = /c0f69897-7f92-43db-ad47-96c3c4721128
14602025-03-07T22:46:54.761ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec zone_address = fd00:1122:3344:101::2
14612025-03-07T22:46:54.761ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
14622025-03-07T22:46:54.761ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "7786af02-3f36-4fe4-92eb-937ccf477a22", "content-length": "26", "date": "Fri, 07 Mar 2025 22:46:54 GMT"} })
14632025-03-07T22:46:54.761ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
14642025-03-07T22:46:54.763ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
14652025-03-07T22:46:54.783ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
14662025-03-07T22:46:54.783ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "d087b8f7-288e-4bf7-b258-bfa070b3ee21", "date": "Fri, 07 Mar 2025 22:46:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
14672025-03-07T22:46:54.783ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
14682025-03-07T22:46:54.786ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "16723341-55de-405d-82d5-c5ec93af24a8", "date": "Fri, 07 Mar 2025 22:46:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
14692025-03-07T22:46:54.786ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
14702025-03-07T22:46:54.787ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
14712025-03-07T22:46:54.787ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 373.048775ms iteration = 16 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
14722025-03-07T22:46:54.823ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
14732025-03-07T22:46:54.823ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d5aef3a-e2c9-481d-b5e2-3f28ff078690", "content-length": "691", "date": "Fri, 07 Mar 2025 22:46:54 GMT"} })
14742025-03-07T22:46:54.823ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
14752025-03-07T22:46:54.823ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
14762025-03-07T22:46:54.823ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
14772025-03-07T22:46:54.825ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 410.98095ms iteration = 27 status = {}
14782025-03-07T22:46:54.833ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
14792025-03-07T22:46:54.834ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c502685-cd11-40cf-9b7b-05ee60afa2ae", "content-length": "2", "date": "Fri, 07 Mar 2025 22:46:54 GMT"} })
14802025-03-07T22:46:54.835ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
14812025-03-07T22:46:54.835ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
14822025-03-07T22:46:54.840ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 425.305497ms iteration = 27 status = {}
14832025-03-07T22:46:54.842ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 427.66933ms iteration = 27 status = null
14842025-03-07T22:46:54.843ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
14852025-03-07T22:46:54.872ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
14862025-03-07T22:46:54.872ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
14872025-03-07T22:46:54.872ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "97ccd14e-8de6-4cac-9596-d0dc20155193", "date": "Fri, 07 Mar 2025 22:46:54 GMT"} })
14882025-03-07T22:46:54.872ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14892025-03-07T22:46:54.872ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14902025-03-07T22:46:54.874ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14912025-03-07T22:46:54.874ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14922025-03-07T22:46:54.877ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14932025-03-07T22:46:54.877ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14942025-03-07T22:46:54.881ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
14952025-03-07T22:46:54.881ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
14962025-03-07T22:46:54.881ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "5e459090-1061-4844-8075-830939fd4521", "date": "Fri, 07 Mar 2025 22:46:54 GMT"} })
14972025-03-07T22:46:54.883ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 468.182359ms iteration = 27 status = {}
14982025-03-07T22:46:54.906ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
14992025-03-07T22:46:54.907ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "100dea3c-63cc-4fcd-824e-647e14d22548", "date": "Fri, 07 Mar 2025 22:46:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
15002025-03-07T22:46:54.907ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
15012025-03-07T22:46:54.909ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "bf34389e-241f-4614-8d39-8d04000ab30b", "date": "Fri, 07 Mar 2025 22:46:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
15022025-03-07T22:46:54.909ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
15032025-03-07T22:46:54.911ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "0b095a20-2008-4de2-8efd-a323afc88dea", "date": "Fri, 07 Mar 2025 22:46:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
15042025-03-07T22:46:54.911ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
15052025-03-07T22:46:54.913ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
15062025-03-07T22:46:54.913ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 498.701883ms iteration = 16 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
15072025-03-07T22:46:54.956ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
15082025-03-07T22:46:54.956ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
15092025-03-07T22:46:54.958ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 2619304f-a0b9-451a-bbfb-b14af6353df6, rack_id: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(8820d7be-fc72-45ff-bc2e-17c4bb9613af) } rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
15102025-03-07T22:46:54.958ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
15112025-03-07T22:46:54.958ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
15122025-03-07T22:46:54.966ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
15132025-03-07T22:46:54.966ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "452bf3a4-8082-4f36-b50f-5573e42bf669", "content-length": "134", "date": "Fri, 07 Mar 2025 22:46:54 GMT"} })
15142025-03-07T22:46:54.966ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
15152025-03-07T22:46:54.966ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 8820d7be-fc72-45ff-bc2e-17c4bb9613af
15162025-03-07T22:46:54.966ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
15172025-03-07T22:46:54.966ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
15182025-03-07T22:46:54.966ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
15192025-03-07T22:46:54.966ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
15202025-03-07T22:46:54.966ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
15212025-03-07T22:46:54.966ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
15222025-03-07T22:46:54.967ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
15232025-03-07T22:46:54.967ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
15242025-03-07T22:46:54.967ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
15252025-03-07T22:46:54.967ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
15262025-03-07T22:46:54.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
15272025-03-07T22:46:54.969ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
15282025-03-07T22:46:54.969ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d3b6cad-0beb-4e46-92d7-1d5ffff3e6ef", "content-length": "2", "date": "Fri, 07 Mar 2025 22:46:54 GMT"} })
15292025-03-07T22:46:54.969ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
15302025-03-07T22:46:54.969ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "19775469-9bf4-42fa-9733-9b1b08656b62", "content-length": "52", "date": "Fri, 07 Mar 2025 22:46:54 GMT"} })
15312025-03-07T22:46:54.969ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
15322025-03-07T22:46:54.969ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
15332025-03-07T22:46:55.068ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "cd13ad96-7836-4cae-8fc0-b8819b162e44", "date": "Fri, 07 Mar 2025 22:46:54 GMT"} })
15342025-03-07T22:46:55.068ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch = Switch0
15352025-03-07T22:46:55.068ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
15362025-03-07T22:46:55.068ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "2846a4d6-6d31-4003-85f6-ca1da2292152", "content-length": "682", "date": "Fri, 07 Mar 2025 22:46:54 GMT"} })
15372025-03-07T22:46:55.068ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
15382025-03-07T22:46:55.069ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
15392025-03-07T22:46:55.073ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
15402025-03-07T22:46:55.073ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
15412025-03-07T22:46:55.076ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
15422025-03-07T22:46:55.077ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
15432025-03-07T22:46:55.079ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
15442025-03-07T22:46:55.079ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
15452025-03-07T22:46:55.086ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 671.941418ms iteration = 27 status = {}
15462025-03-07T22:47:04.415ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 80 reason = Timeout
15472025-03-07T22:47:04.415ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15482025-03-07T22:47:04.415ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 80 reason = Timeout
15492025-03-07T22:47:04.415ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
15502025-03-07T22:47:04.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15512025-03-07T22:47:04.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
15522025-03-07T22:47:04.417ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15532025-03-07T22:47:04.420ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
15542025-03-07T22:47:04.420ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15552025-03-07T22:47:04.420ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15562025-03-07T22:47:04.422ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15572025-03-07T22:47:04.423ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15582025-03-07T22:47:04.424ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15592025-03-07T22:47:04.427ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
15602025-03-07T22:47:04.427ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15612025-03-07T22:47:04.427ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15622025-03-07T22:47:04.439ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
15632025-03-07T22:47:04.439ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 24.816329ms iteration = 80 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
15642025-03-07T22:47:04.447ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
15652025-03-07T22:47:04.447ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
15662025-03-07T22:47:04.448ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 33.607443ms iteration = 80 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
15672025-03-07T22:47:04.681ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 355 local_addr = [fd00:1122:3344:101::c]:41480 method = GET remote_addr = [fd00:1122:3344:101::d]:63523 req_id = c1625734-e765-41d6-a6c7-e7dfc52c03ef response_code = 200 uri = /c0f69897-7f92-43db-ad47-96c3c4721128
15682025-03-07T22:47:14.414ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 81 reason = Timeout
15692025-03-07T22:47:14.414ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 81 reason = Timeout
15702025-03-07T22:47:14.414ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15712025-03-07T22:47:14.414ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
15722025-03-07T22:47:14.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15732025-03-07T22:47:14.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
15742025-03-07T22:47:14.416ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15752025-03-07T22:47:14.419ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
15762025-03-07T22:47:14.419ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15772025-03-07T22:47:14.419ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15782025-03-07T22:47:14.421ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15792025-03-07T22:47:14.421ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15802025-03-07T22:47:14.423ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15812025-03-07T22:47:14.426ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
15822025-03-07T22:47:14.426ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15832025-03-07T22:47:14.426ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15842025-03-07T22:47:14.435ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
15852025-03-07T22:47:14.435ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.537987ms iteration = 81 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
15862025-03-07T22:47:14.445ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
15872025-03-07T22:47:14.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
15882025-03-07T22:47:14.447ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 32.28907ms iteration = 81 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
15892025-03-07T22:47:14.682ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 729 local_addr = [fd00:1122:3344:101::c]:41480 method = GET remote_addr = [fd00:1122:3344:101::d]:63523 req_id = aa19ce0c-331c-4d56-81ec-4ea6d82f758b response_code = 200 uri = /c0f69897-7f92-43db-ad47-96c3c4721128
15902025-03-07T22:47:19.674ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::c]:12221 remote_addr = [fd00:1122:3344:101::d]:60616
15912025-03-07T22:47:19.674ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60616 req_id = 46d40824-5d66-4021-aaca-a93e19c3544b roles = RoleSet { roles: {} } uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?sort_by=id_ascending
15922025-03-07T22:47:19.675ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60616 req_id = 46d40824-5d66-4021-aaca-a93e19c3544b resource = Database result = Ok(()) uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?sort_by=id_ascending
15932025-03-07T22:47:19.678ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3440 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60616 req_id = 46d40824-5d66-4021-aaca-a93e19c3544b response_code = 200 uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?sort_by=id_ascending
15942025-03-07T22:47:19.678ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60616 req_id = c2f34357-dbf4-4f1c-b49d-821b0ded2cdb roles = RoleSet { roles: {} } uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZmUxNjlhMy03NTM2LTRiNTQtOTU5My04ODE1ZDZlODFkZDQifX0%3D
15952025-03-07T22:47:19.678ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60616 req_id = c2f34357-dbf4-4f1c-b49d-821b0ded2cdb resource = Database result = Ok(()) uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZmUxNjlhMy03NTM2LTRiNTQtOTU5My04ODE1ZDZlODFkZDQifX0%3D
15962025-03-07T22:47:19.681ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2601 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60616 req_id = c2f34357-dbf4-4f1c-b49d-821b0ded2cdb response_code = 200 uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZmUxNjlhMy03NTM2LTRiNTQtOTU5My04ODE1ZDZlODFkZDQifX0%3D
15972025-03-07T22:47:24.414ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 30 reason = Timeout
15982025-03-07T22:47:24.415ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 28 reason = Timeout
15992025-03-07T22:47:24.415ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
16002025-03-07T22:47:24.415ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
16012025-03-07T22:47:24.415ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 28 reason = Timeout
16022025-03-07T22:47:24.415ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
16032025-03-07T22:47:24.415ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 28 reason = Timeout
16042025-03-07T22:47:24.415ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
16052025-03-07T22:47:24.415ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 28 reason = Timeout
16062025-03-07T22:47:24.415ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
16072025-03-07T22:47:24.415ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
16082025-03-07T22:47:24.415ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 82 reason = Timeout
16092025-03-07T22:47:24.415ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
16102025-03-07T22:47:24.415ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 28 reason = Timeout
16112025-03-07T22:47:24.415ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 28 reason = Timeout
16122025-03-07T22:47:24.415ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
16132025-03-07T22:47:24.415ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 28 reason = Timeout
16142025-03-07T22:47:24.415ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
16152025-03-07T22:47:24.415ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 28 reason = Timeout
16162025-03-07T22:47:24.415ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
16172025-03-07T22:47:24.415ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 28 reason = Timeout
16182025-03-07T22:47:24.415ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
16192025-03-07T22:47:24.415ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 82 reason = Timeout
16202025-03-07T22:47:24.415ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
16212025-03-07T22:47:24.415ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 28 reason = Timeout
16222025-03-07T22:47:24.415ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
16232025-03-07T22:47:24.415ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16242025-03-07T22:47:24.415ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 28 reason = Timeout
16252025-03-07T22:47:24.415ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
16262025-03-07T22:47:24.415ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 28 reason = Timeout
16272025-03-07T22:47:24.415ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
16282025-03-07T22:47:24.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
16292025-03-07T22:47:24.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
16302025-03-07T22:47:24.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
16312025-03-07T22:47:24.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
16322025-03-07T22:47:24.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
16332025-03-07T22:47:24.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
16342025-03-07T22:47:24.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
16352025-03-07T22:47:24.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
16362025-03-07T22:47:24.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
16372025-03-07T22:47:24.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
16382025-03-07T22:47:24.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
16392025-03-07T22:47:24.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
16402025-03-07T22:47:24.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
16412025-03-07T22:47:24.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
16422025-03-07T22:47:24.416ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 28 reason = Timeout
16432025-03-07T22:47:24.416ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 28 reason = Timeout
16442025-03-07T22:47:24.416ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
16452025-03-07T22:47:24.416ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
16462025-03-07T22:47:24.416ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 28 reason = Timeout
16472025-03-07T22:47:24.416ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
16482025-03-07T22:47:24.416ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 28 reason = Timeout
16492025-03-07T22:47:24.416ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
16502025-03-07T22:47:24.416ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
16512025-03-07T22:47:24.416ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
16522025-03-07T22:47:24.416ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
16532025-03-07T22:47:24.416ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
16542025-03-07T22:47:24.417ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
16552025-03-07T22:47:24.418ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
16562025-03-07T22:47:24.418ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16572025-03-07T22:47:24.419ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.049648ms iteration = 28 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
16582025-03-07T22:47:24.419ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16592025-03-07T22:47:24.419ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16602025-03-07T22:47:24.420ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16612025-03-07T22:47:24.420ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16622025-03-07T22:47:24.421ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
16632025-03-07T22:47:24.421ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
16642025-03-07T22:47:24.421ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
16652025-03-07T22:47:24.422ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
16662025-03-07T22:47:24.422ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
16672025-03-07T22:47:24.422ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
16682025-03-07T22:47:24.422ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16692025-03-07T22:47:24.423ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
16702025-03-07T22:47:24.423ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
16712025-03-07T22:47:24.423ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
16722025-03-07T22:47:24.423ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
16732025-03-07T22:47:24.423ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
16742025-03-07T22:47:24.423ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
16752025-03-07T22:47:24.424ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
16762025-03-07T22:47:24.424ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
16772025-03-07T22:47:24.424ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
16782025-03-07T22:47:24.425ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
16792025-03-07T22:47:24.425ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
16802025-03-07T22:47:24.425ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
16812025-03-07T22:47:24.426ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1602, tv_nsec: 771188429 } })
16822025-03-07T22:47:24.427ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
16832025-03-07T22:47:24.427ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16842025-03-07T22:47:24.427ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
16852025-03-07T22:47:24.427ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
16862025-03-07T22:47:24.427ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
16872025-03-07T22:47:24.427ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1602, tv_nsec: 771877876 } })
16882025-03-07T22:47:24.427ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
16892025-03-07T22:47:24.427ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
16902025-03-07T22:47:24.427ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
16912025-03-07T22:47:24.428ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 11.825264ms iteration = 28 status = {"errors":[],"finish_invoked_ok":[]}
16922025-03-07T22:47:24.428ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
16932025-03-07T22:47:24.429ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
16942025-03-07T22:47:24.429ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
16952025-03-07T22:47:24.429ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
16962025-03-07T22:47:24.429ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
16972025-03-07T22:47:24.431ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
16982025-03-07T22:47:24.431ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16992025-03-07T22:47:24.431ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17002025-03-07T22:47:24.436ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 21.227404ms iteration = 82 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
17012025-03-07T22:47:24.436ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
17022025-03-07T22:47:24.437ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
17032025-03-07T22:47:24.437ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17042025-03-07T22:47:24.439ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 22.849389ms iteration = 28 status = {"errors":[],"requests_created_ok":[]}
17052025-03-07T22:47:24.439ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
17062025-03-07T22:47:24.439ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
17072025-03-07T22:47:24.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
17082025-03-07T22:47:24.440ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
17092025-03-07T22:47:24.440ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
17102025-03-07T22:47:24.440ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
17112025-03-07T22:47:24.441ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
17122025-03-07T22:47:24.441ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
17132025-03-07T22:47:24.441ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
17142025-03-07T22:47:24.441ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
17152025-03-07T22:47:24.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
17162025-03-07T22:47:24.442ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1602, tv_nsec: 786842030 } })
17172025-03-07T22:47:24.442ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
17182025-03-07T22:47:24.445ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
17192025-03-07T22:47:24.445ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17202025-03-07T22:47:24.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17212025-03-07T22:47:24.445ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17222025-03-07T22:47:24.448ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
17232025-03-07T22:47:24.448ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
17242025-03-07T22:47:24.448ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
17252025-03-07T22:47:24.449ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
17262025-03-07T22:47:24.449ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
17272025-03-07T22:47:24.449ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
17282025-03-07T22:47:24.450ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17292025-03-07T22:47:24.451ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
17302025-03-07T22:47:24.452ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
17312025-03-07T22:47:24.453ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17322025-03-07T22:47:24.455ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
17332025-03-07T22:47:24.455ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
17342025-03-07T22:47:24.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17352025-03-07T22:47:24.456ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
17362025-03-07T22:47:24.456ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
17372025-03-07T22:47:24.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
17382025-03-07T22:47:24.458ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17392025-03-07T22:47:24.461ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
17402025-03-07T22:47:24.461ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
17412025-03-07T22:47:24.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
17422025-03-07T22:47:24.470ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
17432025-03-07T22:47:24.470ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
17442025-03-07T22:47:24.471ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
17452025-03-07T22:47:24.471ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 56.801776ms iteration = 82 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
17462025-03-07T22:47:24.472ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
17472025-03-07T22:47:24.472ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
17482025-03-07T22:47:24.474ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 57.612983ms iteration = 28 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
17492025-03-07T22:47:24.474ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
17502025-03-07T22:47:24.474ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
17512025-03-07T22:47:24.476ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
17522025-03-07T22:47:24.476ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
17532025-03-07T22:47:24.478ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17542025-03-07T22:47:24.478ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 61.935705ms iteration = 28 status = {"errors":[],"garbage_collect_requested":[]}
17552025-03-07T22:47:24.480ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 65.479779ms iteration = 28 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
17562025-03-07T22:47:24.481ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 73b218dd-7dbc-4345-acb1-fe69ced4a7f7, lookup_type: ById(73b218dd-7dbc-4345-acb1-fe69ced4a7f7) } result = Ok(())
17572025-03-07T22:47:24.481ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
17582025-03-07T22:47:24.481ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
17592025-03-07T22:47:24.482ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 67.284587ms iteration = 28 status = 0
17602025-03-07T22:47:24.484ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 69.245376ms iteration = 28 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
17612025-03-07T22:47:24.487ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
17622025-03-07T22:47:24.487ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 72.113193ms iteration = 28 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
17632025-03-07T22:47:24.491ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
17642025-03-07T22:47:24.491ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
17652025-03-07T22:47:24.491ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 76.057551ms iteration = 28 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
17662025-03-07T22:47:24.495ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
17672025-03-07T22:47:24.495ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 80.476073ms iteration = 28 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
17682025-03-07T22:47:24.507ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
17692025-03-07T22:47:24.507ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
17702025-03-07T22:47:24.507ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "889949d8-d1ac-46b7-8b86-a9cfca800ca3", "content-length": "26", "date": "Fri, 07 Mar 2025 22:47:24 GMT"} })
17712025-03-07T22:47:24.507ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
17722025-03-07T22:47:24.510ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
17732025-03-07T22:47:24.517ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
17742025-03-07T22:47:24.517ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
17752025-03-07T22:47:24.517ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "c190e06e-c9d2-450b-bd1a-e00e5959b793", "content-length": "26", "date": "Fri, 07 Mar 2025 22:47:24 GMT"} })
17762025-03-07T22:47:24.517ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
17772025-03-07T22:47:24.520ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec zone_address = fd00:1122:3344:101::2
17782025-03-07T22:47:24.520ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
17792025-03-07T22:47:24.520ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
17802025-03-07T22:47:24.520ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8abe73c-5220-495c-8dba-416833e71b8a", "content-length": "26", "date": "Fri, 07 Mar 2025 22:47:24 GMT"} })
17812025-03-07T22:47:24.520ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
17822025-03-07T22:47:24.521ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
17832025-03-07T22:47:24.543ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 128.524786ms iteration = 30 status = {"physical_disks_added":0}
17842025-03-07T22:47:24.660ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 245.509299ms iteration = 28 status = {}
17852025-03-07T22:47:24.662ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
17862025-03-07T22:47:24.662ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
17872025-03-07T22:47:24.663ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c70b079-cebc-4c6d-ad5a-0f9778f7331a", "content-length": "372", "date": "Fri, 07 Mar 2025 22:47:24 GMT"} })
17882025-03-07T22:47:24.663ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
17892025-03-07T22:47:24.663ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
17902025-03-07T22:47:24.681ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 310 local_addr = [fd00:1122:3344:101::c]:41480 method = GET remote_addr = [fd00:1122:3344:101::d]:63523 req_id = 08b1dc52-0312-43e5-87b2-314dc89ad830 response_code = 200 uri = /c0f69897-7f92-43db-ad47-96c3c4721128
17912025-03-07T22:47:24.738ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
17922025-03-07T22:47:24.738ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "0348f7e0-d29e-43cc-905d-a9452ac1ddb3", "content-length": "2", "date": "Fri, 07 Mar 2025 22:47:24 GMT"} })
17932025-03-07T22:47:24.739ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
17942025-03-07T22:47:24.740ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
17952025-03-07T22:47:24.742ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
17962025-03-07T22:47:24.742ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "67f98908-7f42-4fd3-94fb-c007e85274cc", "content-length": "691", "date": "Fri, 07 Mar 2025 22:47:24 GMT"} })
17972025-03-07T22:47:24.742ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
17982025-03-07T22:47:24.742ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
17992025-03-07T22:47:24.742ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
18002025-03-07T22:47:24.744ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 329.340715ms iteration = 28 status = {}
18012025-03-07T22:47:24.745ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 330.008982ms iteration = 28 status = null
18022025-03-07T22:47:24.772ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
18032025-03-07T22:47:24.772ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
18042025-03-07T22:47:24.774ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 2619304f-a0b9-451a-bbfb-b14af6353df6, rack_id: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(8820d7be-fc72-45ff-bc2e-17c4bb9613af) } rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
18052025-03-07T22:47:24.774ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
18062025-03-07T22:47:24.774ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
18072025-03-07T22:47:24.784ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
18082025-03-07T22:47:24.784ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "11ad422b-fa10-4d33-9065-6ff402547525", "content-length": "134", "date": "Fri, 07 Mar 2025 22:47:24 GMT"} })
18092025-03-07T22:47:24.784ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
18102025-03-07T22:47:24.784ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 8820d7be-fc72-45ff-bc2e-17c4bb9613af
18112025-03-07T22:47:24.784ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
18122025-03-07T22:47:24.784ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
18132025-03-07T22:47:24.784ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
18142025-03-07T22:47:24.784ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
18152025-03-07T22:47:24.784ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
18162025-03-07T22:47:24.784ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
18172025-03-07T22:47:24.784ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
18182025-03-07T22:47:24.784ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
18192025-03-07T22:47:24.784ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
18202025-03-07T22:47:24.784ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
18212025-03-07T22:47:24.784ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "09a94516-d5a7-4322-bc1f-9163ad996684", "date": "Fri, 07 Mar 2025 22:47:24 GMT"} })
18222025-03-07T22:47:24.784ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
18232025-03-07T22:47:24.785ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
18242025-03-07T22:47:24.785ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
18252025-03-07T22:47:24.785ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
18262025-03-07T22:47:24.785ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
18272025-03-07T22:47:24.787ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
18282025-03-07T22:47:24.787ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "660dc59b-2f78-4b84-9cc4-76f552d33cc9", "content-length": "2", "date": "Fri, 07 Mar 2025 22:47:24 GMT"} })
18292025-03-07T22:47:24.787ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
18302025-03-07T22:47:24.787ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "069471d4-f1c8-4545-a35a-fbbe528e965b", "content-length": "52", "date": "Fri, 07 Mar 2025 22:47:24 GMT"} })
18312025-03-07T22:47:24.787ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
18322025-03-07T22:47:24.787ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
18332025-03-07T22:47:24.788ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
18342025-03-07T22:47:24.788ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
18352025-03-07T22:47:24.791ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
18362025-03-07T22:47:24.791ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
18372025-03-07T22:47:24.793ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
18382025-03-07T22:47:24.793ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
18392025-03-07T22:47:24.883ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "9a8cbeb3-ebef-422c-b0be-4dc389f06f53", "date": "Fri, 07 Mar 2025 22:47:24 GMT"} })
18402025-03-07T22:47:24.883ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch = Switch0
18412025-03-07T22:47:24.883ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
18422025-03-07T22:47:24.883ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "0cdf98c7-53c2-41e6-9a4d-c65b118b6dd0", "date": "Fri, 07 Mar 2025 22:47:24 GMT"} })
18432025-03-07T22:47:24.885ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 469.956286ms iteration = 28 status = {}
18442025-03-07T22:47:24.885ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f50878b-2519-42d4-bb09-8912135ed4fe", "content-length": "682", "date": "Fri, 07 Mar 2025 22:47:24 GMT"} })
18452025-03-07T22:47:24.885ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
18462025-03-07T22:47:24.885ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
18472025-03-07T22:47:24.888ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
18482025-03-07T22:47:24.888ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
18492025-03-07T22:47:24.891ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
18502025-03-07T22:47:24.892ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
18512025-03-07T22:47:24.894ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
18522025-03-07T22:47:24.894ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
18532025-03-07T22:47:24.900ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 485.600027ms iteration = 28 status = {}
18542025-03-07T22:47:25.194ZDEBGnexus (producer-registration-task): registering / renewing oximeter producer lease with Nexus
18552025-03-07T22:47:25.194ZDEBGnexus (producer-registration-task): using nexus address for registration addr = [fd00:1122:3344:101::c]:12221
18562025-03-07T22:47:25.225ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::c]:12221/metrics/producers
18572025-03-07T22:47:25.225ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::c]:12221 remote_addr = [fd00:1122:3344:101::c]:54930
18582025-03-07T22:47:25.225ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::c]:54930 req_id = 767c3350-26c4-4594-9d50-17fe027fd3d6 roles = RoleSet { roles: {} } uri = /metrics/producers
18592025-03-07T22:47:25.225ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::c]:54930 req_id = 767c3350-26c4-4594-9d50-17fe027fd3d6 resource = Database result = Ok(()) uri = /metrics/producers
18602025-03-07T22:47:25.272ZINFOnexus: assigned collector to new producer collector_id = 7b5e9a24-6b39-4a58-a275-8cc5a75195d8 file = nexus/src/app/oximeter.rs:76 producer_id = c0f69897-7f92-43db-ad47-96c3c4721128
18612025-03-07T22:47:25.272ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 47466 local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::c]:54930 req_id = 767c3350-26c4-4594-9d50-17fe027fd3d6 response_code = 201 uri = /metrics/producers
18622025-03-07T22:47:25.272ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "767c3350-26c4-4594-9d50-17fe027fd3d6", "content-length": "41", "date": "Fri, 07 Mar 2025 22:47:25 GMT"} })
18632025-03-07T22:47:25.274ZDEBGnexus (producer-registration-task): registered with nexus successfully
18642025-03-07T22:47:25.274ZDEBGnexus (producer-registration-task): pausing until time to renew lease lease_duration = 600s wait_period = 150s
18652025-03-07T22:47:34.415ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 83 reason = Timeout
18662025-03-07T22:47:34.415ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 83 reason = Timeout
18672025-03-07T22:47:34.415ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
18682025-03-07T22:47:34.415ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18692025-03-07T22:47:34.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
18702025-03-07T22:47:34.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
18712025-03-07T22:47:34.417ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18722025-03-07T22:47:34.418ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
18732025-03-07T22:47:34.418ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
18742025-03-07T22:47:34.419ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
18752025-03-07T22:47:34.419ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18762025-03-07T22:47:34.420ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
18772025-03-07T22:47:34.422ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18782025-03-07T22:47:34.422ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
18792025-03-07T22:47:34.424ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18802025-03-07T22:47:34.426ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
18812025-03-07T22:47:34.427ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18822025-03-07T22:47:34.427ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
18832025-03-07T22:47:34.438ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
18842025-03-07T22:47:34.438ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.89023ms iteration = 83 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
18852025-03-07T22:47:34.445ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 30.314451ms iteration = 83 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
18862025-03-07T22:47:34.674ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60616 req_id = 2211d4d7-e368-4d3c-a730-da0780e11d07 roles = RoleSet { roles: {} } uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?sort_by=id_ascending
18872025-03-07T22:47:34.675ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60616 req_id = 2211d4d7-e368-4d3c-a730-da0780e11d07 resource = Database result = Ok(()) uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?sort_by=id_ascending
18882025-03-07T22:47:34.677ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3215 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60616 req_id = 2211d4d7-e368-4d3c-a730-da0780e11d07 response_code = 200 uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?sort_by=id_ascending
18892025-03-07T22:47:34.678ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60616 req_id = 9c3b2de2-27a2-4a3c-b913-421279431e85 roles = RoleSet { roles: {} } uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZmUxNjlhMy03NTM2LTRiNTQtOTU5My04ODE1ZDZlODFkZDQifX0%3D
18902025-03-07T22:47:34.678ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60616 req_id = 9c3b2de2-27a2-4a3c-b913-421279431e85 resource = Database result = Ok(()) uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZmUxNjlhMy03NTM2LTRiNTQtOTU5My04ODE1ZDZlODFkZDQifX0%3D
18912025-03-07T22:47:34.680ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2347 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60616 req_id = 9c3b2de2-27a2-4a3c-b913-421279431e85 response_code = 200 uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZmUxNjlhMy03NTM2LTRiNTQtOTU5My04ODE1ZDZlODFkZDQifX0%3D
18922025-03-07T22:47:34.681ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 289 local_addr = [fd00:1122:3344:101::c]:41480 method = GET remote_addr = [fd00:1122:3344:101::d]:63523 req_id = 30d1c07a-bf5a-453d-bfcf-bbd180eb8432 response_code = 200 uri = /c0f69897-7f92-43db-ad47-96c3c4721128
18932025-03-07T22:47:44.415ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 84 reason = Timeout
18942025-03-07T22:47:44.415ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 84 reason = Timeout
18952025-03-07T22:47:44.415ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18962025-03-07T22:47:44.415ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
18972025-03-07T22:47:44.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
18982025-03-07T22:47:44.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
18992025-03-07T22:47:44.417ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19002025-03-07T22:47:44.420ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
19012025-03-07T22:47:44.420ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
19022025-03-07T22:47:44.420ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
19032025-03-07T22:47:44.422ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
19042025-03-07T22:47:44.422ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
19052025-03-07T22:47:44.424ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19062025-03-07T22:47:44.427ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
19072025-03-07T22:47:44.427ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
19082025-03-07T22:47:44.427ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
19092025-03-07T22:47:44.442ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
19102025-03-07T22:47:44.442ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 26.852938ms iteration = 84 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
19112025-03-07T22:47:44.450ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
19122025-03-07T22:47:44.450ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
19132025-03-07T22:47:44.479ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 63.953974ms iteration = 84 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
19142025-03-07T22:47:44.682ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 485 local_addr = [fd00:1122:3344:101::c]:41480 method = GET remote_addr = [fd00:1122:3344:101::d]:63523 req_id = 21e5008a-a76f-4134-9b87-37ac71172ff9 response_code = 200 uri = /c0f69897-7f92-43db-ad47-96c3c4721128
19152025-03-07T22:47:54.415ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 31 reason = Timeout
19162025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
19172025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 16 reason = Timeout
19182025-03-07T22:47:54.416ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f
19192025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 19.86\u{b5}s iteration = 16 status = {"enabled":false,"target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f"}
19202025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 29 reason = Timeout
19212025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 29 reason = Timeout
19222025-03-07T22:47:54.416ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
19232025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
19242025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
19252025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 85 reason = Timeout
19262025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 17 reason = Timeout
19272025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 15 reason = Timeout
19282025-03-07T22:47:54.416ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-07 22:37:54.416053972 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
19292025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
19302025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
19312025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 29 reason = Timeout
19322025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
19332025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 29 reason = Timeout
19342025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
19352025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 29 reason = Timeout
19362025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
19372025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
19382025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 29 reason = Timeout
19392025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
19402025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 15 reason = Timeout
19412025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
19422025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 15 reason = Timeout
19432025-03-07T22:47:54.416ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
19442025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
19452025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 85 reason = Timeout
19462025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
19472025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 29 reason = Timeout
19482025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
19492025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 15 reason = Timeout
19502025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
19512025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 29 reason = Timeout
19522025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
19532025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 29 reason = Timeout
19542025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
19552025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 15 reason = Timeout
19562025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
19572025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 29 reason = Timeout
19582025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
19592025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 15 reason = Timeout
19602025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
19612025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
19622025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 15 reason = Timeout
19632025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 17 reason = Timeout
19642025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 15 reason = Timeout
19652025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
19662025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 15 reason = Timeout
19672025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 29 reason = Timeout
19682025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 29 reason = Timeout
19692025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 15 reason = Timeout
19702025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
19712025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
19722025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
19732025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 29 reason = Timeout
19742025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
19752025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 29 reason = Timeout
19762025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
19772025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 29 reason = Timeout
19782025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
19792025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 29 reason = Timeout
19802025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
19812025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
19822025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
19832025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
19842025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
19852025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
19862025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
19872025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
19882025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
19892025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
19902025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
19912025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
19922025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
19932025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
19942025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
19952025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
19962025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
19972025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
19982025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
19992025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
20002025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
20012025-03-07T22:47:54.416ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("732b0e55-5ccf-4068-b771-1299f872f618.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("fbf58a59-609b-4fb3-b792-2c09f3f7b776.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("fbf58a59-609b-4fb3-b792-2c09f3f7b776.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 1632, tv_nsec: 760901481 } })
20022025-03-07T22:47:54.416ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("114e494f-6e94-4e0c-80db-5823b226f1a3.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("a9773e0b-15d4-4efa-b977-9f3598685cdd.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("d2f04f38-7ac2-4327-a087-c2a47865841a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("d2f04f38-7ac2-4327-a087-c2a47865841a.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:1::1))) }], valid_until: Instant { tv_sec: 1632, tv_nsec: 760897731 } })
20032025-03-07T22:47:54.416ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
20042025-03-07T22:47:54.417ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
20052025-03-07T22:47:54.417ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
20062025-03-07T22:47:54.417ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
20072025-03-07T22:47:54.417ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
20082025-03-07T22:47:54.417ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
20092025-03-07T22:47:54.417ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
20102025-03-07T22:47:54.417ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.189041ms iteration = 15 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
20112025-03-07T22:47:54.417ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
20122025-03-07T22:47:54.417ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.292483ms iteration = 15 status = {"addresses":["[fd00:1122:3344:3::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353"]}
20132025-03-07T22:47:54.419ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20142025-03-07T22:47:54.420ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20152025-03-07T22:47:54.420ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.12182ms iteration = 29 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
20162025-03-07T22:47:54.420ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20172025-03-07T22:47:54.421ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20182025-03-07T22:47:54.421ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
20192025-03-07T22:47:54.421ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
20202025-03-07T22:47:54.421ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
20212025-03-07T22:47:54.422ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 5.849486ms iteration = 15 status = {"errors":[],"found_port_ok":[]}
20222025-03-07T22:47:54.423ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
20232025-03-07T22:47:54.423ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
20242025-03-07T22:47:54.423ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
20252025-03-07T22:47:54.423ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1632, tv_nsec: 767935428 } })
20262025-03-07T22:47:54.424ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
20272025-03-07T22:47:54.424ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
20282025-03-07T22:47:54.424ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
20292025-03-07T22:47:54.424ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
20302025-03-07T22:47:54.425ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
20312025-03-07T22:47:54.425ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
20322025-03-07T22:47:54.425ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
20332025-03-07T22:47:54.426ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20342025-03-07T22:47:54.427ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20352025-03-07T22:47:54.427ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-07 22:37:54.416053972 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
20362025-03-07T22:47:54.428ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 11.938845ms iteration = 15 status = {"expiration":"2025-03-07T22:37:54.416053972Z","pruned":[]}
20372025-03-07T22:47:54.428ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
20382025-03-07T22:47:54.428ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
20392025-03-07T22:47:54.428ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
20402025-03-07T22:47:54.430ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
20412025-03-07T22:47:54.430ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
20422025-03-07T22:47:54.430ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
20432025-03-07T22:47:54.430ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
20442025-03-07T22:47:54.430ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
20452025-03-07T22:47:54.430ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20462025-03-07T22:47:54.431ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20472025-03-07T22:47:54.433ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
20482025-03-07T22:47:54.433ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
20492025-03-07T22:47:54.433ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
20502025-03-07T22:47:54.433ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
20512025-03-07T22:47:54.433ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
20522025-03-07T22:47:54.433ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
20532025-03-07T22:47:54.434ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
20542025-03-07T22:47:54.434ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
20552025-03-07T22:47:54.434ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
20562025-03-07T22:47:54.434ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
20572025-03-07T22:47:54.434ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
20582025-03-07T22:47:54.434ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
20592025-03-07T22:47:54.436ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
20602025-03-07T22:47:54.436ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
20612025-03-07T22:47:54.436ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
20622025-03-07T22:47:54.437ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
20632025-03-07T22:47:54.442ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
20642025-03-07T22:47:54.442ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
20652025-03-07T22:47:54.442ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 25.674797ms iteration = 15 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
20662025-03-07T22:47:54.442ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1632, tv_nsec: 786351355 } })
20672025-03-07T22:47:54.442ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
20682025-03-07T22:47:54.442ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
20692025-03-07T22:47:54.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
20702025-03-07T22:47:54.443ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
20712025-03-07T22:47:54.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
20722025-03-07T22:47:54.445ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
20732025-03-07T22:47:54.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
20742025-03-07T22:47:54.446ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
20752025-03-07T22:47:54.446ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
20762025-03-07T22:47:54.446ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20772025-03-07T22:47:54.451ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
20782025-03-07T22:47:54.452ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1632, tv_nsec: 796189080 } })
20792025-03-07T22:47:54.454ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
20802025-03-07T22:47:54.455ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20812025-03-07T22:47:54.457ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
20822025-03-07T22:47:54.457ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
20832025-03-07T22:47:54.457ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
20842025-03-07T22:47:54.457ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
20852025-03-07T22:47:54.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
20862025-03-07T22:47:54.460ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
20872025-03-07T22:47:54.462ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20882025-03-07T22:47:54.467ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
20892025-03-07T22:47:54.467ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
20902025-03-07T22:47:54.467ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
20912025-03-07T22:47:54.470ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20922025-03-07T22:47:54.474ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
20932025-03-07T22:47:54.474ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
20942025-03-07T22:47:54.474ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
20952025-03-07T22:47:54.482ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20962025-03-07T22:47:54.486ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
20972025-03-07T22:47:54.486ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
20982025-03-07T22:47:54.487ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
20992025-03-07T22:47:54.490ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
21002025-03-07T22:47:54.490ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
21012025-03-07T22:47:54.492ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
21022025-03-07T22:47:54.492ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = a49f17b2-1834-4c40-bc6e-ff0588e4b3ef dns_zone_name = oxide.test found_so_far = 0 version = 2
21032025-03-07T22:47:54.492ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
21042025-03-07T22:47:54.492ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
21052025-03-07T22:47:54.494ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 78.521905ms iteration = 29 status = {"errors":[],"finish_invoked_ok":[]}
21062025-03-07T22:47:54.495ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21072025-03-07T22:47:54.496ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
21082025-03-07T22:47:54.497ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
21092025-03-07T22:47:54.498ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 82.465632ms iteration = 29 status = {"errors":[],"garbage_collect_requested":[]}
21102025-03-07T22:47:54.499ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
21112025-03-07T22:47:54.499ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
21122025-03-07T22:47:54.499ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 82.930597ms iteration = 85 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
21132025-03-07T22:47:54.499ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
21142025-03-07T22:47:54.501ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
21152025-03-07T22:47:54.501ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21162025-03-07T22:47:54.501ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
21172025-03-07T22:47:54.502ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
21182025-03-07T22:47:54.502ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
21192025-03-07T22:47:54.502ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 85.960196ms iteration = 15 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
21202025-03-07T22:47:54.502ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 85.713574ms iteration = 29 status = {"errors":[],"requests_created_ok":[]}
21212025-03-07T22:47:54.502ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
21222025-03-07T22:47:54.502ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
21232025-03-07T22:47:54.504ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
21242025-03-07T22:47:54.504ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 88.719762ms iteration = 85 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
21252025-03-07T22:47:54.504ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
21262025-03-07T22:47:54.505ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
21272025-03-07T22:47:54.508ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
21282025-03-07T22:47:54.508ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
21292025-03-07T22:47:54.508ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
21302025-03-07T22:47:54.510ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21312025-03-07T22:47:54.510ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21322025-03-07T22:47:54.511ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
21332025-03-07T22:47:54.511ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
21342025-03-07T22:47:54.511ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
21352025-03-07T22:47:54.511ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
21362025-03-07T22:47:54.511ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 95.149124ms iteration = 29 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
21372025-03-07T22:47:54.511ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
21382025-03-07T22:47:54.513ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
21392025-03-07T22:47:54.513ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
21402025-03-07T22:47:54.514ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
21412025-03-07T22:47:54.515ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 73b218dd-7dbc-4345-acb1-fe69ced4a7f7, lookup_type: ById(73b218dd-7dbc-4345-acb1-fe69ced4a7f7) } result = Ok(())
21422025-03-07T22:47:54.515ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
21432025-03-07T22:47:54.515ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = a49f17b2-1834-4c40-bc6e-ff0588e4b3ef dns_zone_name = oxide.test found_so_far = 1 version = 2
21442025-03-07T22:47:54.515ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
21452025-03-07T22:47:54.515ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-07 22:34:17.797135 UTC dns_group = external generation = 2
21462025-03-07T22:47:54.515ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 99.006492ms iteration = 15 status = {"generation":2}
21472025-03-07T22:47:54.515ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
21482025-03-07T22:47:54.517ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21492025-03-07T22:47:54.519ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
21502025-03-07T22:47:54.519ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
21512025-03-07T22:47:54.520ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
21522025-03-07T22:47:54.520ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
21532025-03-07T22:47:54.520ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
21542025-03-07T22:47:54.525ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 108.900856ms iteration = 29 status = 0
21552025-03-07T22:47:54.525ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
21562025-03-07T22:47:54.525ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
21572025-03-07T22:47:54.527ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
21582025-03-07T22:47:54.529ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
21592025-03-07T22:47:54.529ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
21602025-03-07T22:47:54.529ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
21612025-03-07T22:47:54.529ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
21622025-03-07T22:47:54.533ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
21632025-03-07T22:47:54.533ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = fdc30f7a-36a7-4893-91d6-3456da7362c0 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
21642025-03-07T22:47:54.533ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
21652025-03-07T22:47:54.534ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21662025-03-07T22:47:54.534ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
21672025-03-07T22:47:54.536ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
21682025-03-07T22:47:54.536ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
21692025-03-07T22:47:54.536ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
21702025-03-07T22:47:54.546ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 130.214341ms iteration = 29 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
21712025-03-07T22:47:54.548ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21722025-03-07T22:47:54.548ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
21732025-03-07T22:47:54.549ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
21742025-03-07T22:47:54.550ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
21752025-03-07T22:47:54.550ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
21762025-03-07T22:47:54.550ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
21772025-03-07T22:47:54.550ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
21782025-03-07T22:47:54.560ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
21792025-03-07T22:47:54.560ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 144.58816ms iteration = 29 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
21802025-03-07T22:47:54.573ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 156.885798ms iteration = 29 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
21812025-03-07T22:47:54.583ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
21822025-03-07T22:47:54.583ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
21832025-03-07T22:47:54.584ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c5dc485-5865-4b6e-a0ce-fa6da0e839c4", "content-length": "26", "date": "Fri, 07 Mar 2025 22:47:54 GMT"} })
21842025-03-07T22:47:54.584ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
21852025-03-07T22:47:54.585ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
21862025-03-07T22:47:54.585ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
21872025-03-07T22:47:54.586ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
21882025-03-07T22:47:54.587ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
21892025-03-07T22:47:54.587ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
21902025-03-07T22:47:54.588ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b07f7d3-9f22-4ebe-aa40-06c9df3a43c1", "content-length": "26", "date": "Fri, 07 Mar 2025 22:47:54 GMT"} })
21912025-03-07T22:47:54.588ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
21922025-03-07T22:47:54.591ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
21932025-03-07T22:47:54.614ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec zone_address = fd00:1122:3344:101::2
21942025-03-07T22:47:54.614ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
21952025-03-07T22:47:54.615ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ee10c4e-1476-4ecd-ae92-1cbf84c5cb0e", "content-length": "26", "date": "Fri, 07 Mar 2025 22:47:54 GMT"} })
21962025-03-07T22:47:54.615ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
21972025-03-07T22:47:54.617ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
21982025-03-07T22:47:54.679ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
21992025-03-07T22:47:54.679ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
22002025-03-07T22:47:54.680ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22012025-03-07T22:47:54.682ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22022025-03-07T22:47:54.682ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 350 local_addr = [fd00:1122:3344:101::c]:41480 method = GET remote_addr = [fd00:1122:3344:101::d]:63523 req_id = 7283e580-3438-4952-93b5-0d01deeb15d8 response_code = 200 uri = /c0f69897-7f92-43db-ad47-96c3c4721128
22032025-03-07T22:47:54.682ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
22042025-03-07T22:47:54.682ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
22052025-03-07T22:47:54.683ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
22062025-03-07T22:47:54.686ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
22072025-03-07T22:47:54.686ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
22082025-03-07T22:47:54.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
22092025-03-07T22:47:54.689ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = fdc30f7a-36a7-4893-91d6-3456da7362c0 dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
22102025-03-07T22:47:54.689ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
22112025-03-07T22:47:54.689ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-07 22:34:17.208516 UTC dns_group = internal generation = 1
22122025-03-07T22:47:54.689ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 273.351047ms iteration = 15 status = {"generation":1}
22132025-03-07T22:47:54.718ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
22142025-03-07T22:47:54.728ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 312.14421ms iteration = 31 status = {"physical_disks_added":0}
22152025-03-07T22:47:54.730ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
22162025-03-07T22:47:54.730ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 314.579113ms iteration = 29 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
22172025-03-07T22:47:54.731ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
22182025-03-07T22:47:54.731ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
22192025-03-07T22:47:54.731ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 315.30169ms iteration = 29 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
22202025-03-07T22:47:54.733ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-07T22:33:52.171499Z, time_modified: 2025-03-07T22:33:52.171499Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, db_silo: Silo { identity: SiloIdentity { id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-07T22:34:17.695388Z, time_modified: 2025-03-07T22:34:17.695388Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, db_silo: Silo { identity: SiloIdentity { id: ebb5815c-a32b-49c0-a621-3a7cd3eb3452, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-07T22:34:17.695388Z, time_modified: 2025-03-07T22:34:17.695388Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1" }] }) }
22212025-03-07T22:47:54.733ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 317.247369ms iteration = 15 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"ebb5815c-a32b-49c0-a621-3a7cd3eb3452","tls_certs":["f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1"]}},"default_endpoint":{"silo_id":"ebb5815c-a32b-49c0-a621-3a7cd3eb3452","tls_certs":["f857482e363fc36f169844018fd10beb52cce480a27bbdb173d6c9c730aad5e1"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
22222025-03-07T22:47:54.735ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
22232025-03-07T22:47:54.735ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "4cca8221-a761-4adf-a36a-75ca47c6b7f4", "date": "Fri, 07 Mar 2025 22:47:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
22242025-03-07T22:47:54.735ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
22252025-03-07T22:47:54.737ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "5c7578a5-863d-4bdf-bd42-b799a470f201", "date": "Fri, 07 Mar 2025 22:47:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
22262025-03-07T22:47:54.738ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
22272025-03-07T22:47:54.739ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
22282025-03-07T22:47:54.739ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 323.073475ms iteration = 17 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
22292025-03-07T22:47:54.766ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 350.218815ms iteration = 29 status = {}
22302025-03-07T22:47:54.777ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
22312025-03-07T22:47:54.777ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 360.778347ms iteration = 15 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
22322025-03-07T22:47:54.784ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
22332025-03-07T22:47:54.784ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
22342025-03-07T22:47:54.788ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "c03be751-6690-43f5-8a38-351e04bc6d78", "content-length": "372", "date": "Fri, 07 Mar 2025 22:47:54 GMT"} })
22352025-03-07T22:47:54.788ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
22362025-03-07T22:47:54.789ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
22372025-03-07T22:47:54.803ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
22382025-03-07T22:47:54.803ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
22392025-03-07T22:47:54.803ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "e4e3a1ce-79d0-4520-8806-9ae9f6efe344", "date": "Fri, 07 Mar 2025 22:47:54 GMT"} })
22402025-03-07T22:47:54.803ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
22412025-03-07T22:47:54.804ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
22422025-03-07T22:47:54.806ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
22432025-03-07T22:47:54.806ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
22442025-03-07T22:47:54.809ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
22452025-03-07T22:47:54.809ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
22462025-03-07T22:47:54.813ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
22472025-03-07T22:47:54.813ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
22482025-03-07T22:47:54.813ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "bda35444-8d42-4a1e-8882-1af3fb26de9a", "date": "Fri, 07 Mar 2025 22:47:54 GMT"} })
22492025-03-07T22:47:54.814ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 398.774622ms iteration = 29 status = {}
22502025-03-07T22:47:54.814ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22512025-03-07T22:47:54.816ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
22522025-03-07T22:47:54.816ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
22532025-03-07T22:47:54.816ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
22542025-03-07T22:47:54.818ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22552025-03-07T22:47:54.819ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
22562025-03-07T22:47:54.819ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
22572025-03-07T22:47:54.820ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
22582025-03-07T22:47:54.835ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
22592025-03-07T22:47:54.835ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "9bf6764f-a265-41df-b79f-65041f353ab0", "date": "Fri, 07 Mar 2025 22:47:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
22602025-03-07T22:47:54.835ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
22612025-03-07T22:47:54.837ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "03f3c3c5-4d00-40b1-9e71-3068fb139201", "date": "Fri, 07 Mar 2025 22:47:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
22622025-03-07T22:47:54.837ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
22632025-03-07T22:47:54.838ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "0a290328-c68f-4940-8c1d-273f26047802", "date": "Fri, 07 Mar 2025 22:47:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
22642025-03-07T22:47:54.838ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
22652025-03-07T22:47:54.840ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:3::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353] }
22662025-03-07T22:47:54.840ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 424.56944ms iteration = 17 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
22672025-03-07T22:47:54.847ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
22682025-03-07T22:47:54.847ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "4040f2af-17f2-43d0-aac7-b828eb11c640", "content-length": "2", "date": "Fri, 07 Mar 2025 22:47:54 GMT"} })
22692025-03-07T22:47:54.849ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
22702025-03-07T22:47:54.849ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
22712025-03-07T22:47:54.851ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 435.586636ms iteration = 29 status = null
22722025-03-07T22:47:54.890ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
22732025-03-07T22:47:54.890ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "c57c22b5-45b1-43d3-9ec9-e9dbfae147c3", "content-length": "691", "date": "Fri, 07 Mar 2025 22:47:54 GMT"} })
22742025-03-07T22:47:54.890ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
22752025-03-07T22:47:54.890ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
22762025-03-07T22:47:54.890ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
22772025-03-07T22:47:54.893ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 477.093705ms iteration = 29 status = {}
22782025-03-07T22:47:54.931ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
22792025-03-07T22:47:54.931ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
22802025-03-07T22:47:54.933ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 2619304f-a0b9-451a-bbfb-b14af6353df6, rack_id: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(8820d7be-fc72-45ff-bc2e-17c4bb9613af) } rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
22812025-03-07T22:47:54.933ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
22822025-03-07T22:47:54.933ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
22832025-03-07T22:47:54.943ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
22842025-03-07T22:47:54.943ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "1abb269b-5596-4db8-aed8-b03f3d0b2c92", "content-length": "134", "date": "Fri, 07 Mar 2025 22:47:54 GMT"} })
22852025-03-07T22:47:54.943ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
22862025-03-07T22:47:54.943ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 8820d7be-fc72-45ff-bc2e-17c4bb9613af
22872025-03-07T22:47:54.943ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
22882025-03-07T22:47:54.943ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
22892025-03-07T22:47:54.943ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
22902025-03-07T22:47:54.943ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
22912025-03-07T22:47:54.943ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
22922025-03-07T22:47:54.943ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
22932025-03-07T22:47:54.943ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
22942025-03-07T22:47:54.943ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
22952025-03-07T22:47:54.944ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
22962025-03-07T22:47:54.944ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
22972025-03-07T22:47:54.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
22982025-03-07T22:47:54.946ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
22992025-03-07T22:47:54.946ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "6bc3731a-f611-46b7-94f8-09d434825a3d", "content-length": "2", "date": "Fri, 07 Mar 2025 22:47:54 GMT"} })
23002025-03-07T22:47:54.946ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
23012025-03-07T22:47:54.946ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "29255ed4-9029-45eb-a46b-2dff9d358e51", "content-length": "52", "date": "Fri, 07 Mar 2025 22:47:54 GMT"} })
23022025-03-07T22:47:54.946ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
23032025-03-07T22:47:54.946ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
23042025-03-07T22:47:55.046ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "19699fc5-26e5-48a0-a152-51c906ec4a97", "date": "Fri, 07 Mar 2025 22:47:54 GMT"} })
23052025-03-07T22:47:55.046ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch = Switch0
23062025-03-07T22:47:55.046ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
23072025-03-07T22:47:55.046ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2692d33-517b-483c-bb48-33aeecd40ba1", "content-length": "682", "date": "Fri, 07 Mar 2025 22:47:54 GMT"} })
23082025-03-07T22:47:55.046ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
23092025-03-07T22:47:55.046ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
23102025-03-07T22:47:55.050ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
23112025-03-07T22:47:55.050ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
23122025-03-07T22:47:55.053ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
23132025-03-07T22:47:55.054ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
23142025-03-07T22:47:55.056ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
23152025-03-07T22:47:55.056ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
23162025-03-07T22:47:55.062ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 646.17772ms iteration = 29 status = {}
23172025-03-07T22:48:04.416ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 86 reason = Timeout
23182025-03-07T22:48:04.416ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 86 reason = Timeout
23192025-03-07T22:48:04.416ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
23202025-03-07T22:48:04.416ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
23212025-03-07T22:48:04.416ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
23222025-03-07T22:48:04.417ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
23232025-03-07T22:48:04.419ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23242025-03-07T22:48:04.421ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
23252025-03-07T22:48:04.421ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
23262025-03-07T22:48:04.422ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
23272025-03-07T22:48:04.424ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
23282025-03-07T22:48:04.424ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
23292025-03-07T22:48:04.426ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23302025-03-07T22:48:04.428ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
23312025-03-07T22:48:04.428ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
23322025-03-07T22:48:04.429ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
23332025-03-07T22:48:04.438ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
23342025-03-07T22:48:04.438ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.89525ms iteration = 86 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
23352025-03-07T22:48:04.447ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
23362025-03-07T22:48:04.447ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
23372025-03-07T22:48:04.483ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 66.900253ms iteration = 86 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
23382025-03-07T22:48:04.674ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60616 req_id = 8b030c77-efa0-4cea-828e-7961674d8479 roles = RoleSet { roles: {} } uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?sort_by=id_ascending
23392025-03-07T22:48:04.675ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60616 req_id = 8b030c77-efa0-4cea-828e-7961674d8479 resource = Database result = Ok(()) uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?sort_by=id_ascending
23402025-03-07T22:48:04.677ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2921 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60616 req_id = 8b030c77-efa0-4cea-828e-7961674d8479 response_code = 200 uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?sort_by=id_ascending
23412025-03-07T22:48:04.677ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60616 req_id = 56b892b6-442d-42df-887a-6962445c9a53 roles = RoleSet { roles: {} } uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZmUxNjlhMy03NTM2LTRiNTQtOTU5My04ODE1ZDZlODFkZDQifX0%3D
23422025-03-07T22:48:04.678ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60616 req_id = 56b892b6-442d-42df-887a-6962445c9a53 resource = Database result = Ok(()) uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZmUxNjlhMy03NTM2LTRiNTQtOTU5My04ODE1ZDZlODFkZDQifX0%3D
23432025-03-07T22:48:04.680ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2444 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:60616 req_id = 56b892b6-442d-42df-887a-6962445c9a53 response_code = 200 uri = /metrics/collectors/7b5e9a24-6b39-4a58-a275-8cc5a75195d8/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJkZmUxNjlhMy03NTM2LTRiNTQtOTU5My04ODE1ZDZlODFkZDQifX0%3D
23442025-03-07T22:48:04.682ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 288 local_addr = [fd00:1122:3344:101::c]:41480 method = GET remote_addr = [fd00:1122:3344:101::d]:63523 req_id = d4d7d2ae-02b5-4643-bbfc-1aae33205d8a response_code = 200 uri = /c0f69897-7f92-43db-ad47-96c3c4721128
23452025-03-07T22:48:14.416ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 87 reason = Timeout
23462025-03-07T22:48:14.416ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 87 reason = Timeout
23472025-03-07T22:48:14.416ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
23482025-03-07T22:48:14.416ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
23492025-03-07T22:48:14.416ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
23502025-03-07T22:48:14.416ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
23512025-03-07T22:48:14.417ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23522025-03-07T22:48:14.418ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
23532025-03-07T22:48:14.418ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
23542025-03-07T22:48:14.420ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.018569ms iteration = 87 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
23552025-03-07T22:48:14.420ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
23562025-03-07T22:48:14.420ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
23572025-03-07T22:48:14.420ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
23582025-03-07T22:48:14.422ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
23592025-03-07T22:48:14.422ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
23602025-03-07T22:48:14.423ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23612025-03-07T22:48:14.426ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
23622025-03-07T22:48:14.426ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
23632025-03-07T22:48:14.426ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
23642025-03-07T22:48:14.435ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
23652025-03-07T22:48:14.435ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 18.999973ms iteration = 87 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
23662025-03-07T22:48:14.682ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 549 local_addr = [fd00:1122:3344:101::c]:41480 method = GET remote_addr = [fd00:1122:3344:101::d]:63523 req_id = 4ea09802-a983-4e04-8069-9b0077681f17 response_code = 200 uri = /c0f69897-7f92-43db-ad47-96c3c4721128
23672025-03-07T22:48:24.416ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 32 reason = Timeout
23682025-03-07T22:48:24.416ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
23692025-03-07T22:48:24.416ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 30 reason = Timeout
23702025-03-07T22:48:24.416ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
23712025-03-07T22:48:24.416ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 30 reason = Timeout
23722025-03-07T22:48:24.416ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 88 reason = Timeout
23732025-03-07T22:48:24.416ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
23742025-03-07T22:48:24.416ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 30 reason = Timeout
23752025-03-07T22:48:24.416ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
23762025-03-07T22:48:24.416ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
23772025-03-07T22:48:24.416ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 30 reason = Timeout
23782025-03-07T22:48:24.416ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
23792025-03-07T22:48:24.416ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
23802025-03-07T22:48:24.416ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 30 reason = Timeout
23812025-03-07T22:48:24.416ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
23822025-03-07T22:48:24.416ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 30 reason = Timeout
23832025-03-07T22:48:24.416ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
23842025-03-07T22:48:24.416ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 30 reason = Timeout
23852025-03-07T22:48:24.416ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
23862025-03-07T22:48:24.416ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 30 reason = Timeout
23872025-03-07T22:48:24.416ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
23882025-03-07T22:48:24.416ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
23892025-03-07T22:48:24.416ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 30 reason = Timeout
23902025-03-07T22:48:24.416ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
23912025-03-07T22:48:24.416ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 30 reason = Timeout
23922025-03-07T22:48:24.416ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
23932025-03-07T22:48:24.416ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 30 reason = Timeout
23942025-03-07T22:48:24.416ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
23952025-03-07T22:48:24.416ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 88 reason = Timeout
23962025-03-07T22:48:24.416ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 30 reason = Timeout
23972025-03-07T22:48:24.416ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
23982025-03-07T22:48:24.416ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
23992025-03-07T22:48:24.416ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
24002025-03-07T22:48:24.416ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
24012025-03-07T22:48:24.416ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
24022025-03-07T22:48:24.416ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24032025-03-07T22:48:24.416ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
24042025-03-07T22:48:24.416ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
24052025-03-07T22:48:24.416ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
24062025-03-07T22:48:24.416ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
24072025-03-07T22:48:24.416ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
24082025-03-07T22:48:24.416ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
24092025-03-07T22:48:24.416ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24102025-03-07T22:48:24.416ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
24112025-03-07T22:48:24.416ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
24122025-03-07T22:48:24.417ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 30 reason = Timeout
24132025-03-07T22:48:24.417ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 30 reason = Timeout
24142025-03-07T22:48:24.417ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 30 reason = Timeout
24152025-03-07T22:48:24.417ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
24162025-03-07T22:48:24.417ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
24172025-03-07T22:48:24.417ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
24182025-03-07T22:48:24.417ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 30 reason = Timeout
24192025-03-07T22:48:24.417ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
24202025-03-07T22:48:24.417ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
24212025-03-07T22:48:24.417ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
24222025-03-07T22:48:24.417ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
24232025-03-07T22:48:24.417ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
24242025-03-07T22:48:24.419ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.523274ms iteration = 30 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
24252025-03-07T22:48:24.420ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24262025-03-07T22:48:24.422ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1662, tv_nsec: 765533475 } })
24272025-03-07T22:48:24.422ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24282025-03-07T22:48:24.422ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
24292025-03-07T22:48:24.423ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
24302025-03-07T22:48:24.423ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24312025-03-07T22:48:24.423ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24322025-03-07T22:48:24.425ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24332025-03-07T22:48:24.425ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24342025-03-07T22:48:24.427ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
24352025-03-07T22:48:24.427ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
24362025-03-07T22:48:24.427ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
24372025-03-07T22:48:24.427ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
24382025-03-07T22:48:24.427ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
24392025-03-07T22:48:24.427ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
24402025-03-07T22:48:24.427ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
24412025-03-07T22:48:24.427ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
24422025-03-07T22:48:24.428ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
24432025-03-07T22:48:24.429ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1662, tv_nsec: 773045438 } })
24442025-03-07T22:48:24.429ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
24452025-03-07T22:48:24.430ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 12.913384ms iteration = 30 status = {"errors":[],"finish_invoked_ok":[]}
24462025-03-07T22:48:24.432ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
24472025-03-07T22:48:24.432ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
24482025-03-07T22:48:24.432ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-73b218dd-7dbc-4345-acb1-fe69ced4a7f7.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1662, tv_nsec: 776135077 } })
24492025-03-07T22:48:24.432ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
24502025-03-07T22:48:24.453ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24512025-03-07T22:48:24.454ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
24522025-03-07T22:48:24.454ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
24532025-03-07T22:48:24.456ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
24542025-03-07T22:48:24.456ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
24552025-03-07T22:48:24.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
24562025-03-07T22:48:24.456ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 40.895583ms iteration = 88 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
24572025-03-07T22:48:24.458ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
24582025-03-07T22:48:24.458ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
24592025-03-07T22:48:24.458ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24602025-03-07T22:48:24.458ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
24612025-03-07T22:48:24.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
24622025-03-07T22:48:24.460ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
24632025-03-07T22:48:24.460ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
24642025-03-07T22:48:24.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
24652025-03-07T22:48:24.461ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
24662025-03-07T22:48:24.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
24672025-03-07T22:48:24.463ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
24682025-03-07T22:48:24.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
24692025-03-07T22:48:24.466ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 48.976721ms iteration = 30 status = {"errors":[],"garbage_collect_requested":[]}
24702025-03-07T22:48:24.469ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
24712025-03-07T22:48:24.469ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 52.422664ms iteration = 30 status = {"errors":[],"requests_created_ok":[]}
24722025-03-07T22:48:24.470ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
24732025-03-07T22:48:24.471ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
24742025-03-07T22:48:24.471ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
24752025-03-07T22:48:24.472ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 56.068109ms iteration = 30 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
24762025-03-07T22:48:24.474ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
24772025-03-07T22:48:24.474ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
24782025-03-07T22:48:24.476ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24792025-03-07T22:48:24.477ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
24802025-03-07T22:48:24.477ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
24812025-03-07T22:48:24.477ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
24822025-03-07T22:48:24.477ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
24832025-03-07T22:48:24.479ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
24842025-03-07T22:48:24.479ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24852025-03-07T22:48:24.479ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24862025-03-07T22:48:24.483ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24872025-03-07T22:48:24.483ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24882025-03-07T22:48:24.484ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24892025-03-07T22:48:24.487ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
24902025-03-07T22:48:24.487ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24912025-03-07T22:48:24.487ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24922025-03-07T22:48:24.489ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24932025-03-07T22:48:24.491ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
24942025-03-07T22:48:24.491ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 75.545746ms iteration = 30 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
24952025-03-07T22:48:24.492ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
24962025-03-07T22:48:24.492ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
24972025-03-07T22:48:24.492ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
24982025-03-07T22:48:24.501ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
24992025-03-07T22:48:24.501ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 85.295539ms iteration = 88 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
25002025-03-07T22:48:24.504ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
25012025-03-07T22:48:24.504ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
25022025-03-07T22:48:24.504ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "90420e01-35d4-4be3-8f1a-004848c93f44", "content-length": "26", "date": "Fri, 07 Mar 2025 22:48:24 GMT"} })
25032025-03-07T22:48:24.504ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
25042025-03-07T22:48:24.504ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 88.49801ms iteration = 30 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
25052025-03-07T22:48:24.505ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25062025-03-07T22:48:24.505ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25072025-03-07T22:48:24.506ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
25082025-03-07T22:48:24.506ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
25092025-03-07T22:48:24.506ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
25102025-03-07T22:48:24.508ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
25112025-03-07T22:48:24.508ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
25122025-03-07T22:48:24.508ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
25132025-03-07T22:48:24.510ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 73b218dd-7dbc-4345-acb1-fe69ced4a7f7, lookup_type: ById(73b218dd-7dbc-4345-acb1-fe69ced4a7f7) } result = Ok(())
25142025-03-07T22:48:24.510ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
25152025-03-07T22:48:24.510ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
25162025-03-07T22:48:24.510ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
25172025-03-07T22:48:24.510ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
25182025-03-07T22:48:24.510ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
25192025-03-07T22:48:24.510ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
25202025-03-07T22:48:24.513ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 96.466697ms iteration = 30 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
25212025-03-07T22:48:24.514ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec zone_address = fd00:1122:3344:101::2
25222025-03-07T22:48:24.514ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
25232025-03-07T22:48:24.515ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "07f32d2b-3b4f-4849-8b61-96b4b69ef6f0", "content-length": "26", "date": "Fri, 07 Mar 2025 22:48:24 GMT"} })
25242025-03-07T22:48:24.515ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
25252025-03-07T22:48:24.516ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
25262025-03-07T22:48:24.533ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
25272025-03-07T22:48:24.533ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
25282025-03-07T22:48:24.538ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25292025-03-07T22:48:24.540ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
25302025-03-07T22:48:24.540ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
25312025-03-07T22:48:24.540ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
25322025-03-07T22:48:24.548ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
25332025-03-07T22:48:24.548ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
25342025-03-07T22:48:24.548ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "608de4c7-7a27-433d-9f28-384e32b2cb30", "content-length": "26", "date": "Fri, 07 Mar 2025 22:48:24 GMT"} })
25352025-03-07T22:48:24.548ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
25362025-03-07T22:48:24.550ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
25372025-03-07T22:48:24.612ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 196.252855ms iteration = 32 status = {"physical_disks_added":0}
25382025-03-07T22:48:24.621ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
25392025-03-07T22:48:24.621ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
25402025-03-07T22:48:24.621ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "ead77064-cd16-448b-9c79-477ce76a8164", "content-length": "372", "date": "Fri, 07 Mar 2025 22:48:24 GMT"} })
25412025-03-07T22:48:24.622ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
25422025-03-07T22:48:24.622ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
25432025-03-07T22:48:24.624ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 207.966079ms iteration = 30 status = 0
25442025-03-07T22:48:24.651ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 235.536533ms iteration = 30 status = {}
25452025-03-07T22:48:24.652ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
25462025-03-07T22:48:24.652ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 236.588854ms iteration = 30 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
25472025-03-07T22:48:24.655ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
25482025-03-07T22:48:24.655ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
25492025-03-07T22:48:24.656ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 239.932836ms iteration = 30 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
25502025-03-07T22:48:24.666ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
25512025-03-07T22:48:24.666ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
25522025-03-07T22:48:24.666ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "7c209dfc-8dc1-4603-b7a9-2dbdddd06e07", "date": "Fri, 07 Mar 2025 22:48:24 GMT"} })
25532025-03-07T22:48:24.666ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
25542025-03-07T22:48:24.667ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
25552025-03-07T22:48:24.669ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
25562025-03-07T22:48:24.669ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
25572025-03-07T22:48:24.672ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
25582025-03-07T22:48:24.672ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
25592025-03-07T22:48:24.676ZINFOnexus (BackgroundTasks): VPC route manager sled 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
25602025-03-07T22:48:24.676ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
25612025-03-07T22:48:24.676ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "5e8f164b-3c24-4cf1-8db2-5ebeb3f05286", "date": "Fri, 07 Mar 2025 22:48:24 GMT"} })
25622025-03-07T22:48:24.678ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 261.907667ms iteration = 30 status = {}
25632025-03-07T22:48:24.682ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 308 local_addr = [fd00:1122:3344:101::c]:41480 method = GET remote_addr = [fd00:1122:3344:101::d]:63523 req_id = 1f96773b-b14b-463d-8c32-90db51dde26b response_code = 200 uri = /c0f69897-7f92-43db-ad47-96c3c4721128
25642025-03-07T22:48:24.715ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
25652025-03-07T22:48:24.716ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
25662025-03-07T22:48:24.717ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25672025-03-07T22:48:24.719ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
25682025-03-07T22:48:24.719ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
25692025-03-07T22:48:24.719ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
25702025-03-07T22:48:24.721ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25712025-03-07T22:48:24.722ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
25722025-03-07T22:48:24.722ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
25732025-03-07T22:48:24.722ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
25742025-03-07T22:48:24.732ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
25752025-03-07T22:48:24.733ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "d2314c8a-f3c2-4205-9500-d5f9d7586291", "content-length": "2", "date": "Fri, 07 Mar 2025 22:48:24 GMT"} })
25762025-03-07T22:48:24.734ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
25772025-03-07T22:48:24.735ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
25782025-03-07T22:48:24.736ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 320.55785ms iteration = 30 status = null
25792025-03-07T22:48:24.787ZDEBGnexus (BackgroundTasks): client request SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
25802025-03-07T22:48:24.787ZDEBGnexus (BackgroundTasks): client response SledAgent = 73b218dd-7dbc-4345-acb1-fe69ced4a7f7 background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "125f69d5-da0c-4c57-b890-22a5c6daf99b", "content-length": "691", "date": "Fri, 07 Mar 2025 22:48:24 GMT"} })
25812025-03-07T22:48:24.787ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
25822025-03-07T22:48:24.787ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
25832025-03-07T22:48:24.787ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
25842025-03-07T22:48:24.789ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 373.759682ms iteration = 30 status = {}
25852025-03-07T22:48:24.811ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
25862025-03-07T22:48:24.811ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
25872025-03-07T22:48:24.814ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 2619304f-a0b9-451a-bbfb-b14af6353df6, rack_id: 82e3cdb7-0f87-46cf-98cd-832919b5c7ec, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(8820d7be-fc72-45ff-bc2e-17c4bb9613af) } rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
25882025-03-07T22:48:24.814ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
25892025-03-07T22:48:24.814ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
25902025-03-07T22:48:24.823ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
25912025-03-07T22:48:24.823ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "c05b25b0-2ba9-47c5-8f82-7f467e977da5", "content-length": "134", "date": "Fri, 07 Mar 2025 22:48:24 GMT"} })
25922025-03-07T22:48:24.823ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
25932025-03-07T22:48:24.823ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 8820d7be-fc72-45ff-bc2e-17c4bb9613af
25942025-03-07T22:48:24.823ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
25952025-03-07T22:48:24.823ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
25962025-03-07T22:48:24.823ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
25972025-03-07T22:48:24.823ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
25982025-03-07T22:48:24.823ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
25992025-03-07T22:48:24.823ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
26002025-03-07T22:48:24.823ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
26012025-03-07T22:48:24.823ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
26022025-03-07T22:48:24.824ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec
26032025-03-07T22:48:24.824ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
26042025-03-07T22:48:24.824ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
26052025-03-07T22:48:24.825ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
26062025-03-07T22:48:24.825ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e06e7a0-ae0b-43f3-b9c0-fc7818ef51e6", "content-length": "2", "date": "Fri, 07 Mar 2025 22:48:24 GMT"} })
26072025-03-07T22:48:24.825ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
26082025-03-07T22:48:24.825ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "a776e3cb-9069-43ba-95a8-932c7365ba17", "content-length": "52", "date": "Fri, 07 Mar 2025 22:48:24 GMT"} })
26092025-03-07T22:48:24.825ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch_location = Switch0
26102025-03-07T22:48:24.826ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
26112025-03-07T22:48:24.922ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "bebe15b4-a3a9-47ee-a468-8f3400eea64a", "date": "Fri, 07 Mar 2025 22:48:24 GMT"} })
26122025-03-07T22:48:24.922ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec switch = Switch0
26132025-03-07T22:48:24.922ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
26142025-03-07T22:48:24.922ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 82e3cdb7-0f87-46cf-98cd-832919b5c7ec result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "6035becf-ea76-4381-8bcf-9f9834bc5df6", "content-length": "682", "date": "Fri, 07 Mar 2025 22:48:24 GMT"} })
26152025-03-07T22:48:24.922ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
26162025-03-07T22:48:24.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
26172025-03-07T22:48:24.925ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
26182025-03-07T22:48:24.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
26192025-03-07T22:48:24.928ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
26202025-03-07T22:48:24.929ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
26212025-03-07T22:48:24.931ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
26222025-03-07T22:48:24.931ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
26232025-03-07T22:48:24.937ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 521.825715ms iteration = 30 status = {}
26242025-03-07T22:48:34.416ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 89 reason = Timeout
26252025-03-07T22:48:34.416ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 89 reason = Timeout
26262025-03-07T22:48:34.416ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
26272025-03-07T22:48:34.416ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
26282025-03-07T22:48:34.417ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
26292025-03-07T22:48:34.417ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
26302025-03-07T22:48:34.419ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26312025-03-07T22:48:34.422ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
26322025-03-07T22:48:34.422ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
26332025-03-07T22:48:34.422ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
26342025-03-07T22:48:34.424ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
26352025-03-07T22:48:34.425ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
26362025-03-07T22:48:34.450ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
26372025-03-07T22:48:34.450ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
26382025-03-07T22:48:34.457ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26392025-03-07T22:48:34.459ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
26402025-03-07T22:48:34.459ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
26412025-03-07T22:48:34.459ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
26422025-03-07T22:48:34.474ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
26432025-03-07T22:48:34.474ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 57.561724ms iteration = 89 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
26442025-03-07T22:48:34.485ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 69.133075ms iteration = 89 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
26452025-03-07T22:48:34.682ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 316 local_addr = [fd00:1122:3344:101::c]:41480 method = GET remote_addr = [fd00:1122:3344:101::d]:63523 req_id = 59ec0d6e-fd7e-426f-b3d5-a6460778f187 response_code = 200 uri = /c0f69897-7f92-43db-ad47-96c3c4721128
26462025-03-07T22:48:44.417ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 90 reason = Timeout
26472025-03-07T22:48:44.417ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 90 reason = Timeout
26482025-03-07T22:48:44.417ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
26492025-03-07T22:48:44.417ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
26502025-03-07T22:48:44.417ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
26512025-03-07T22:48:44.417ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
26522025-03-07T22:48:44.448ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
26532025-03-07T22:48:44.448ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26542025-03-07T22:48:44.448ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
26552025-03-07T22:48:44.450ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
26562025-03-07T22:48:44.450ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
26572025-03-07T22:48:44.451ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
26582025-03-07T22:48:44.453ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
26592025-03-07T22:48:44.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
26602025-03-07T22:48:44.455ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26612025-03-07T22:48:44.457ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f, lookup_type: ById(0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f) } result = Ok(())
26622025-03-07T22:48:44.457ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
26632025-03-07T22:48:44.457ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
26642025-03-07T22:48:44.468ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f original_time_created = 2025-03-07 22:33:48.706097 UTC target_id = 0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f time_created = 2025-03-07 22:33:48.706097 UTC
26652025-03-07T22:48:44.468ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 51.937959ms iteration = 90 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"0f6b2d08-1e6c-46f8-a3ec-6b18ce7a0d3f","time_created":"2025-03-07T22:33:48.706097Z"}
26662025-03-07T22:48:44.478ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 61.257989ms iteration = 90 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
26672025-03-07T22:48:44.683ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 305 local_addr = [fd00:1122:3344:101::c]:41480 method = GET remote_addr = [fd00:1122:3344:101::d]:63523 req_id = a6f8b9b7-65a0-4087-9a61-b14f5f0e6d14 response_code = 200 uri = /c0f69897-7f92-43db-ad47-96c3c4721128