Max level shown:
12025-03-21T02:15:06.922ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 65 reason = Timeout
22025-03-21T02:15:06.922ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
32025-03-21T02:15:06.922ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 65 reason = Timeout
42025-03-21T02:15:06.922ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
52025-03-21T02:15:06.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
62025-03-21T02:15:06.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
72025-03-21T02:15:06.924ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
82025-03-21T02:15:06.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
92025-03-21T02:15:06.925ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102025-03-21T02:15:06.926ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.117491ms iteration = 65 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
112025-03-21T02:15:06.928ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
122025-03-21T02:15:06.928ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
132025-03-21T02:15:06.928ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
142025-03-21T02:15:06.930ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
152025-03-21T02:15:06.930ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
162025-03-21T02:15:06.932ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
172025-03-21T02:15:06.934ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
182025-03-21T02:15:06.934ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
192025-03-21T02:15:06.934ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
202025-03-21T02:15:06.941ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
212025-03-21T02:15:06.941ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 19.285096ms iteration = 65 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
222025-03-21T02:15:09.723ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 306 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = e4063af2-3512-4831-8deb-416bd91f33ec response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
232025-03-21T02:15:16.922ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 66 reason = Timeout
242025-03-21T02:15:16.922ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
252025-03-21T02:15:16.922ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 66 reason = Timeout
262025-03-21T02:15:16.922ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
272025-03-21T02:15:16.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
282025-03-21T02:15:16.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
292025-03-21T02:15:16.925ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
302025-03-21T02:15:16.928ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
312025-03-21T02:15:16.928ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
322025-03-21T02:15:16.928ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
332025-03-21T02:15:16.930ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
342025-03-21T02:15:16.930ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
352025-03-21T02:15:16.932ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
362025-03-21T02:15:16.934ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
372025-03-21T02:15:16.934ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
382025-03-21T02:15:16.935ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
392025-03-21T02:15:16.942ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
402025-03-21T02:15:16.942ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 19.504229ms iteration = 66 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
412025-03-21T02:15:16.953ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
422025-03-21T02:15:16.953ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
432025-03-21T02:15:16.981ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 59.001217ms iteration = 66 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
442025-03-21T02:15:19.723ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 270 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = 0fe8915c-6a97-4c9f-86b9-d795c5b2741b response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
452025-03-21T02:15:24.714ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::c]:12221 remote_addr = [fd00:1122:3344:101::d]:57035
462025-03-21T02:15:24.715ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:57035 req_id = 60ab1605-7dcd-45af-a50f-37d2bca8b7d4 roles = RoleSet { roles: {} } uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?sort_by=id_ascending
472025-03-21T02:15:24.715ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:57035 req_id = 60ab1605-7dcd-45af-a50f-37d2bca8b7d4 resource = Database result = Ok(()) uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?sort_by=id_ascending
482025-03-21T02:15:24.717ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2980 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:57035 req_id = 60ab1605-7dcd-45af-a50f-37d2bca8b7d4 response_code = 200 uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?sort_by=id_ascending
492025-03-21T02:15:24.718ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:57035 req_id = 55c60f5d-2325-4598-8325-3eb297784289 roles = RoleSet { roles: {} } uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlMDVlYjc5OS04Y2YwLTQ3MDItYmQzYi05OTk0YWU4NjExOGEifX0%3D
502025-03-21T02:15:24.718ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:57035 req_id = 55c60f5d-2325-4598-8325-3eb297784289 resource = Database result = Ok(()) uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlMDVlYjc5OS04Y2YwLTQ3MDItYmQzYi05OTk0YWU4NjExOGEifX0%3D
512025-03-21T02:15:24.721ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2949 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:57035 req_id = 55c60f5d-2325-4598-8325-3eb297784289 response_code = 200 uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlMDVlYjc5OS04Y2YwLTQ3MDItYmQzYi05OTk0YWU4NjExOGEifX0%3D
522025-03-21T02:15:26.922ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 67 reason = Timeout
532025-03-21T02:15:26.922ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
542025-03-21T02:15:26.922ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 23 reason = Timeout
552025-03-21T02:15:26.922ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
562025-03-21T02:15:26.922ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 23 reason = Timeout
572025-03-21T02:15:26.922ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 23 reason = Timeout
582025-03-21T02:15:26.922ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
592025-03-21T02:15:26.922ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 14 reason = Timeout
602025-03-21T02:15:26.922ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 12 reason = Timeout
612025-03-21T02:15:26.922ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 12 reason = Timeout
622025-03-21T02:15:26.922ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
632025-03-21T02:15:26.922ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 12 reason = Timeout
642025-03-21T02:15:26.922ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 23 reason = Timeout
652025-03-21T02:15:26.922ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 23 reason = Timeout
662025-03-21T02:15:26.922ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
672025-03-21T02:15:26.922ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
682025-03-21T02:15:26.922ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 12 reason = Timeout
692025-03-21T02:15:26.922ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
702025-03-21T02:15:26.922ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
712025-03-21T02:15:26.922ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 23 reason = Timeout
722025-03-21T02:15:26.923ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
732025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 25 reason = Timeout
742025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
752025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
762025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 23 reason = Timeout
772025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 23 reason = Timeout
782025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 23 reason = Timeout
792025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
802025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
812025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
822025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 13 reason = Timeout
832025-03-21T02:15:26.923ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074
842025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 23 reason = Timeout
852025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
862025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 12 reason = Timeout
872025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 15.31\u{b5}s iteration = 13 status = {"enabled":false,"target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074"}
882025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
892025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
902025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
912025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 23 reason = Timeout
922025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 23 reason = Timeout
932025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
942025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
952025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 23 reason = Timeout
962025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
972025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 23 reason = Timeout
982025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
992025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 67 reason = Timeout
1002025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 14 reason = Timeout
1012025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
1022025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 12 reason = Timeout
1032025-03-21T02:15:26.923ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
1042025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
1052025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 12 reason = Timeout
1062025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
1072025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 12 reason = Timeout
1082025-03-21T02:15:26.923ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-21 02:05:26.923124997 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
1092025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
1102025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 12 reason = Timeout
1112025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
1122025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 23 reason = Timeout
1132025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
1142025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
1152025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 12 reason = Timeout
1162025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 23 reason = Timeout
1172025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
1182025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1192025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
1202025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
1212025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
1222025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
1232025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
1242025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
1252025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
1262025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
1272025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
1282025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
1292025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
1302025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
1312025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
1322025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
1332025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
1342025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
1352025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1362025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
1372025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
1382025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
1392025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
1402025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
1412025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
1422025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
1432025-03-21T02:15:26.923ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("a1777e88-1a9c-49b1-bca1-ec6c5e1991fc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("bfbad16c-d6a4-4742-9db2-139d724c0771.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("bfbad16c-d6a4-4742-9db2-139d724c0771.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 1635, tv_nsec: 542858134 } })
1442025-03-21T02:15:26.923ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("200fb1b3-8e37-4e66-8f46-93219f5750f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("4999f8c7-3dfb-459a-ba6c-e61aa5543438.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("91dd1310-1f23-4802-b96e-b07c13e4ea67.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("91dd1310-1f23-4802-b96e-b07c13e4ea67.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:3::1))) }], valid_until: Instant { tv_sec: 1635, tv_nsec: 542862184 } })
1452025-03-21T02:15:26.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
1462025-03-21T02:15:26.924ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
1472025-03-21T02:15:26.924ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.518086ms iteration = 12 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
1482025-03-21T02:15:26.924ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } dns_group = internal
1492025-03-21T02:15:26.924ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.566527ms iteration = 12 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353","[fd00:1122:3344:3::1]:5353"]}
1502025-03-21T02:15:26.925ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
1512025-03-21T02:15:26.926ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
1522025-03-21T02:15:26.926ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.50683ms iteration = 23 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
1532025-03-21T02:15:26.928ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
1542025-03-21T02:15:26.928ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
1552025-03-21T02:15:26.928ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
1562025-03-21T02:15:26.928ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 5.695369ms iteration = 12 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
1572025-03-21T02:15:26.928ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
1582025-03-21T02:15:26.928ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1592025-03-21T02:15:26.928ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1635, tv_nsec: 547745709 } })
1602025-03-21T02:15:26.928ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
1612025-03-21T02:15:26.928ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
1622025-03-21T02:15:26.928ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
1632025-03-21T02:15:26.930ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1642025-03-21T02:15:26.931ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
1652025-03-21T02:15:26.931ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
1662025-03-21T02:15:26.931ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
1672025-03-21T02:15:26.931ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
1682025-03-21T02:15:26.931ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
1692025-03-21T02:15:26.933ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
1702025-03-21T02:15:26.933ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
1712025-03-21T02:15:26.933ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
1722025-03-21T02:15:26.933ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
1732025-03-21T02:15:26.933ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
1742025-03-21T02:15:26.934ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1752025-03-21T02:15:26.935ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
1762025-03-21T02:15:26.935ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
1772025-03-21T02:15:26.935ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
1782025-03-21T02:15:26.935ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
1792025-03-21T02:15:26.935ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1635, tv_nsec: 554950575 } })
1802025-03-21T02:15:26.936ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
1812025-03-21T02:15:26.936ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
1822025-03-21T02:15:26.936ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
1832025-03-21T02:15:26.936ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
1842025-03-21T02:15:26.937ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1852025-03-21T02:15:26.937ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1862025-03-21T02:15:26.938ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1872025-03-21T02:15:26.940ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
1882025-03-21T02:15:26.940ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1892025-03-21T02:15:26.941ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
1902025-03-21T02:15:26.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1912025-03-21T02:15:26.941ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
1922025-03-21T02:15:26.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
1932025-03-21T02:15:26.941ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
1942025-03-21T02:15:26.941ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
1952025-03-21T02:15:26.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
1962025-03-21T02:15:26.943ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 20.713961ms iteration = 12 status = {"errors":[],"found_port_ok":[]}
1972025-03-21T02:15:26.954ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1982025-03-21T02:15:26.954ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 31.740383ms iteration = 67 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
1992025-03-21T02:15:26.956ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
2002025-03-21T02:15:26.957ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
2012025-03-21T02:15:26.957ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
2022025-03-21T02:15:26.961ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-21 02:05:26.923124997 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
2032025-03-21T02:15:26.961ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 38.416139ms iteration = 12 status = {"expiration":"2025-03-21T02:05:26.923124997Z","pruned":[]}
2042025-03-21T02:15:26.963ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
2052025-03-21T02:15:26.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
2062025-03-21T02:15:26.965ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2072025-03-21T02:15:26.965ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
2082025-03-21T02:15:26.965ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
2092025-03-21T02:15:26.966ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
2102025-03-21T02:15:26.966ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2112025-03-21T02:15:26.966ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1635, tv_nsec: 585858963 } })
2122025-03-21T02:15:26.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
2132025-03-21T02:15:26.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2142025-03-21T02:15:26.967ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
2152025-03-21T02:15:26.968ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
2162025-03-21T02:15:26.971ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
2172025-03-21T02:15:26.971ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
2182025-03-21T02:15:26.975ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2192025-03-21T02:15:26.977ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
2202025-03-21T02:15:26.977ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
2212025-03-21T02:15:26.978ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 54.975417ms iteration = 23 status = {"errors":[],"garbage_collect_requested":[]}
2222025-03-21T02:15:26.978ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
2232025-03-21T02:15:26.980ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2242025-03-21T02:15:26.984ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
2252025-03-21T02:15:26.984ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
2262025-03-21T02:15:26.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
2272025-03-21T02:15:26.986ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2282025-03-21T02:15:26.987ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 64.038015ms iteration = 23 status = {"errors":[],"finish_invoked_ok":[]}
2292025-03-21T02:15:26.988ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
2302025-03-21T02:15:26.988ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
2312025-03-21T02:15:26.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
2322025-03-21T02:15:26.993ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 70.797783ms iteration = 23 status = {"errors":[],"requests_created_ok":[]}
2332025-03-21T02:15:27.002ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
2342025-03-21T02:15:27.002ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
2352025-03-21T02:15:27.003ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
2362025-03-21T02:15:27.003ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
2372025-03-21T02:15:27.004ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2382025-03-21T02:15:27.005ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2392025-03-21T02:15:27.005ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
2402025-03-21T02:15:27.006ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
2412025-03-21T02:15:27.007ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
2422025-03-21T02:15:27.007ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
2432025-03-21T02:15:27.007ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
2442025-03-21T02:15:27.008ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
2452025-03-21T02:15:27.008ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
2462025-03-21T02:15:27.009ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
2472025-03-21T02:15:27.009ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 86.769571ms iteration = 23 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
2482025-03-21T02:15:27.009ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
2492025-03-21T02:15:27.009ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
2502025-03-21T02:15:27.010ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
2512025-03-21T02:15:27.010ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 87.998312ms iteration = 23 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
2522025-03-21T02:15:27.012ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
2532025-03-21T02:15:27.012ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
2542025-03-21T02:15:27.015ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2552025-03-21T02:15:27.016ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
2562025-03-21T02:15:27.016ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
2572025-03-21T02:15:27.016ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
2582025-03-21T02:15:27.016ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
2592025-03-21T02:15:27.018ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2602025-03-21T02:15:27.019ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
2612025-03-21T02:15:27.019ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
2622025-03-21T02:15:27.019ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
2632025-03-21T02:15:27.019ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
2642025-03-21T02:15:27.019ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
2652025-03-21T02:15:27.020ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
2662025-03-21T02:15:27.021ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
2672025-03-21T02:15:27.021ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
2682025-03-21T02:15:27.021ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
2692025-03-21T02:15:27.021ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
2702025-03-21T02:15:27.021ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
2712025-03-21T02:15:27.021ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
2722025-03-21T02:15:27.021ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
2732025-03-21T02:15:27.022ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
2742025-03-21T02:15:27.023ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
2752025-03-21T02:15:27.023ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
2762025-03-21T02:15:27.023ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 100.747934ms iteration = 23 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
2772025-03-21T02:15:27.027ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2782025-03-21T02:15:27.029ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2792025-03-21T02:15:27.031ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
2802025-03-21T02:15:27.031ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
2812025-03-21T02:15:27.031ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
2822025-03-21T02:15:27.033ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2832025-03-21T02:15:27.033ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
2842025-03-21T02:15:27.033ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
2852025-03-21T02:15:27.034ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
2862025-03-21T02:15:27.036ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
2872025-03-21T02:15:27.036ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2882025-03-21T02:15:27.037ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2892025-03-21T02:15:27.044ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
2902025-03-21T02:15:27.044ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
2912025-03-21T02:15:27.046ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2922025-03-21T02:15:27.048ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
2932025-03-21T02:15:27.048ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
2942025-03-21T02:15:27.048ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
2952025-03-21T02:15:27.049ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 126.612754ms iteration = 23 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
2962025-03-21T02:15:27.052ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
2972025-03-21T02:15:27.052ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 129.718739ms iteration = 67 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
2982025-03-21T02:15:27.052ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 129.81558ms iteration = 23 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
2992025-03-21T02:15:27.054ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
3002025-03-21T02:15:27.054ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
3012025-03-21T02:15:27.057ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
3022025-03-21T02:15:27.057ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
3032025-03-21T02:15:27.057ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 134.007003ms iteration = 12 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
3042025-03-21T02:15:27.058ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3052025-03-21T02:15:27.062ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: ad5b28f8-6cad-4221-97f9-e84a46a6381b, lookup_type: ById(ad5b28f8-6cad-4221-97f9-e84a46a6381b) } result = Ok(())
3062025-03-21T02:15:27.062ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
3072025-03-21T02:15:27.062ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
3082025-03-21T02:15:27.065ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 141.996983ms iteration = 23 status = 0
3092025-03-21T02:15:27.070ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 147.313044ms iteration = 25 status = {"physical_disks_added":0}
3102025-03-21T02:15:27.072ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
3112025-03-21T02:15:27.072ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 0481b934-0ed5-44d2-96f2-0a9adc385687 dns_zone_name = oxide.test found_so_far = 0 version = 2
3122025-03-21T02:15:27.072ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
3132025-03-21T02:15:27.072ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
3142025-03-21T02:15:27.074ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3152025-03-21T02:15:27.078ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
3162025-03-21T02:15:27.078ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
3172025-03-21T02:15:27.078ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
3182025-03-21T02:15:27.078ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
3192025-03-21T02:15:27.078ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 78c1ab2b-9e58-4297-a89e-d4fa0c8a69c3 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
3202025-03-21T02:15:27.079ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
3212025-03-21T02:15:27.079ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
3222025-03-21T02:15:27.081ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3232025-03-21T02:15:27.081ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 0481b934-0ed5-44d2-96f2-0a9adc385687 dns_zone_name = oxide.test found_so_far = 1 version = 2
3242025-03-21T02:15:27.081ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
3252025-03-21T02:15:27.081ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-21 02:04:31.662541 UTC dns_group = external generation = 2
3262025-03-21T02:15:27.081ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 158.835035ms iteration = 12 status = {"generation":2}
3272025-03-21T02:15:27.085ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
3282025-03-21T02:15:27.085ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
3292025-03-21T02:15:27.085ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
3302025-03-21T02:15:27.088ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 78c1ab2b-9e58-4297-a89e-d4fa0c8a69c3 dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
3312025-03-21T02:15:27.088ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
3322025-03-21T02:15:27.088ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-21 02:04:30.886360 UTC dns_group = internal generation = 1
3332025-03-21T02:15:27.088ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 165.166276ms iteration = 12 status = {"generation":1}
3342025-03-21T02:15:27.100ZINFOnexus (BackgroundTasks): found Silo/DNS/TLS config (changed) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-21T02:04:22.853740Z, time_modified: 2025-03-21T02:04:22.853740Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 89475719-7d35-403a-9037-042f6e747a73, db_silo: Silo { identity: SiloIdentity { id: 89475719-7d35-403a-9037-042f6e747a73, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-21T02:04:31.619926Z, time_modified: 2025-03-21T02:04:31.619926Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "c6a715a41442cf931d7245f8c6684eb4210f9631513e8dd8f0b24981cd6bd960" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 89475719-7d35-403a-9037-042f6e747a73, db_silo: Silo { identity: SiloIdentity { id: 89475719-7d35-403a-9037-042f6e747a73, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-21T02:04:31.619926Z, time_modified: 2025-03-21T02:04:31.619926Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "c6a715a41442cf931d7245f8c6684eb4210f9631513e8dd8f0b24981cd6bd960" }] }) } file = nexus/src/app/background/tasks/external_endpoints.rs:94
3352025-03-21T02:15:27.100ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 177.009712ms iteration = 12 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"89475719-7d35-403a-9037-042f6e747a73","tls_certs":["c6a715a41442cf931d7245f8c6684eb4210f9631513e8dd8f0b24981cd6bd960"]}},"default_endpoint":{"silo_id":"89475719-7d35-403a-9037-042f6e747a73","tls_certs":["c6a715a41442cf931d7245f8c6684eb4210f9631513e8dd8f0b24981cd6bd960"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
3362025-03-21T02:15:27.102ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
3372025-03-21T02:15:27.102ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
3382025-03-21T02:15:27.102ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "40a09263-3bdc-44e0-b244-dde22959808f", "content-length": "26", "date": "Fri, 21 Mar 2025 02:15:27 GMT"} })
3392025-03-21T02:15:27.102ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
3402025-03-21T02:15:27.104ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
3412025-03-21T02:15:27.110ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
3422025-03-21T02:15:27.110ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
3432025-03-21T02:15:27.115ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3442025-03-21T02:15:27.117ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
3452025-03-21T02:15:27.117ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
3462025-03-21T02:15:27.117ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
3472025-03-21T02:15:27.119ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3482025-03-21T02:15:27.121ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
3492025-03-21T02:15:27.121ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
3502025-03-21T02:15:27.121ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
3512025-03-21T02:15:27.149ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 zone_address = fd00:1122:3344:101::2
3522025-03-21T02:15:27.149ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
3532025-03-21T02:15:27.150ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd7a8a45-bde5-495e-9d96-28e0e9886793", "content-length": "26", "date": "Fri, 21 Mar 2025 02:15:27 GMT"} })
3542025-03-21T02:15:27.150ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
3552025-03-21T02:15:27.152ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
3562025-03-21T02:15:27.172ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
3572025-03-21T02:15:27.172ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
3582025-03-21T02:15:27.172ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c0f07ff-c01b-4664-ae2b-a64381da33a8", "content-length": "26", "date": "Fri, 21 Mar 2025 02:15:27 GMT"} })
3592025-03-21T02:15:27.172ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
3602025-03-21T02:15:27.174ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
3612025-03-21T02:15:27.189ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
3622025-03-21T02:15:27.189ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 266.305357ms iteration = 12 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
3632025-03-21T02:15:27.281ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
3642025-03-21T02:15:27.300ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = instance_watcher body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vmms/ae0209c8-aa90-4d04-9274-a71f5070a43b/state
3652025-03-21T02:15:27.312ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
3662025-03-21T02:15:27.359ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
3672025-03-21T02:15:27.387ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "fbbcdaf3-3291-4854-a98d-193409e54730", "content-length": "810", "date": "Fri, 21 Mar 2025 02:15:27 GMT"} })
3682025-03-21T02:15:27.387ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = instance_watcher result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vmms/ae0209c8-aa90-4d04-9274-a71f5070a43b/state", status: 200, headers: {"content-type": "application/json", "x-request-id": "067bc845-918e-4f7f-be44-4e35fe31e020", "content-length": "130", "date": "Fri, 21 Mar 2025 02:15:27 GMT"} })
3692025-03-21T02:15:27.393ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {VirtualNetworkInterfaceHost { physical_host_ip: fd00:1122:3344:101::1, virtual_ip: 172.30.0.5, virtual_mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), vni: Vni(1297745) }} sled = buskin
3702025-03-21T02:15:27.393ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
3712025-03-21T02:15:27.393ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
3722025-03-21T02:15:27.395ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 472.504115ms iteration = 23 status = {}
3732025-03-21T02:15:27.395ZDEBGnexus (BackgroundTasks): updating instance state background_task = instance_watcher instance_id = b340824b-5792-4c7d-9cef-988b21a2d495 sled_id = ad5b28f8-6cad-4221-97f9-e84a46a6381b state = SledVmmState { vmm_state: VmmRuntimeState { state: Running, gen: Generation(4), time_updated: 2025-03-21T02:15:20.291958818Z }, migration_in: None, migration_out: None } vmm_id = ae0209c8-aa90-4d04-9274-a71f5070a43b
3742025-03-21T02:15:27.395ZINFOnexus (BackgroundTasks): received new VMM runtime state from sled agent background_task = instance_watcher file = nexus/src/app/instance.rs:2088 instance_id = b340824b-5792-4c7d-9cef-988b21a2d495 migration_state = Migrations { migration_in: None, migration_out: None } propolis_id = ae0209c8-aa90-4d04-9274-a71f5070a43b sled_id = ad5b28f8-6cad-4221-97f9-e84a46a6381b vmm_id = ae0209c8-aa90-4d04-9274-a71f5070a43b vmm_state = VmmRuntimeState { state: Running, gen: Generation(4), time_updated: 2025-03-21T02:15:20.291958818Z }
3752025-03-21T02:15:27.395ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher instance_id = b340824b-5792-4c7d-9cef-988b21a2d495 roles = RoleSet { roles: {} } sled_id = ad5b28f8-6cad-4221-97f9-e84a46a6381b vmm_id = ae0209c8-aa90-4d04-9274-a71f5070a43b
3762025-03-21T02:15:27.396ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher instance_id = b340824b-5792-4c7d-9cef-988b21a2d495 resource = Database result = Ok(()) sled_id = ad5b28f8-6cad-4221-97f9-e84a46a6381b vmm_id = ae0209c8-aa90-4d04-9274-a71f5070a43b
3772025-03-21T02:15:27.422ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
3782025-03-21T02:15:27.422ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
3792025-03-21T02:15:27.423ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "08937f68-8825-46ab-bb52-5bd348cc8242", "content-length": "583", "date": "Fri, 21 Mar 2025 02:15:27 GMT"} })
3802025-03-21T02:15:27.423ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
3812025-03-21T02:15:27.423ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
3822025-03-21T02:15:27.440ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 1 total_failed = 0 total_incomplete = 0 total_instances = 1 update_sagas_queued = 0
3832025-03-21T02:15:27.442ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 519.30693ms iteration = 23 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{"running":1},"pruned_instances":0,"total_instances":1,"update_sagas_queued":0}
3842025-03-21T02:15:27.464ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
3852025-03-21T02:15:27.464ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4785e11-1010-475a-b8a4-f1c7d778278f", "content-length": "1", "date": "Fri, 21 Mar 2025 02:15:26 GMT"} })
3862025-03-21T02:15:27.466ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
3872025-03-21T02:15:27.466ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
3882025-03-21T02:15:27.467ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {2a3493eb-a49f-4ebe-a01d-1e25ef18044c: {10.151.1.176: {99caa8fe-904a-4b1f-8feb-aef33e243c7c}, 10.151.1.175: {99caa8fe-904a-4b1f-8feb-aef33e243c7c}}} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
3892025-03-21T02:15:27.467ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
3902025-03-21T02:15:27.468ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "4b5f09cc-f9ee-4630-9dd1-61a341720e9f", "date": "Fri, 21 Mar 2025 02:15:27 GMT"} })
3912025-03-21T02:15:27.468ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
3922025-03-21T02:15:27.468ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
3932025-03-21T02:15:27.468ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 546.175398ms iteration = 23 status = null
3942025-03-21T02:15:27.470ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
3952025-03-21T02:15:27.471ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
3962025-03-21T02:15:27.474ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
3972025-03-21T02:15:27.474ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
3982025-03-21T02:15:27.478ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
3992025-03-21T02:15:27.478ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
4002025-03-21T02:15:27.479ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
4012025-03-21T02:15:27.480ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
4022025-03-21T02:15:27.482ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
4032025-03-21T02:15:27.483ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
4042025-03-21T02:15:27.485ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
4052025-03-21T02:15:27.485ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
4062025-03-21T02:15:27.485ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
4072025-03-21T02:15:27.485ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "91948d7c-4abd-46f8-9a43-6359bcfed023", "date": "Fri, 21 Mar 2025 02:15:27 GMT"} })
4082025-03-21T02:15:27.487ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 564.815762ms iteration = 23 status = {}
4092025-03-21T02:15:27.556ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 633.734772ms iteration = 23 status = {}
4102025-03-21T02:15:27.605ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
4112025-03-21T02:15:27.605ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "d935bb1f-e767-43bc-8a36-ac95dcfa7fd3", "date": "Fri, 21 Mar 2025 02:15:27 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
4122025-03-21T02:15:27.605ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
4132025-03-21T02:15:27.607ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "c7254673-4578-4ef3-b9a6-2b56418da379", "date": "Fri, 21 Mar 2025 02:15:27 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
4142025-03-21T02:15:27.607ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
4152025-03-21T02:15:27.609ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
4162025-03-21T02:15:27.609ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 686.615953ms iteration = 14 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
4172025-03-21T02:15:27.614ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
4182025-03-21T02:15:27.614ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
4192025-03-21T02:15:27.616ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 033eb27d-ce1b-4d9a-9cf5-5fe7e1ba28e0, rack_id: 9e10fa12-3248-42fd-baa3-e5de51ba7745, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(f0ee038c-25a9-431e-b4eb-d29544961ae6) } rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
4202025-03-21T02:15:27.616ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
4212025-03-21T02:15:27.616ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
4222025-03-21T02:15:27.617ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
4232025-03-21T02:15:27.627ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
4242025-03-21T02:15:27.627ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b6a8d87-1838-4417-a56b-be901d4c378d", "content-length": "134", "date": "Fri, 21 Mar 2025 02:15:26 GMT"} })
4252025-03-21T02:15:27.627ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
4262025-03-21T02:15:27.627ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = f0ee038c-25a9-431e-b4eb-d29544961ae6
4272025-03-21T02:15:27.628ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
4282025-03-21T02:15:27.628ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
4292025-03-21T02:15:27.628ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
4302025-03-21T02:15:27.628ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
4312025-03-21T02:15:27.628ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
4322025-03-21T02:15:27.628ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
4332025-03-21T02:15:27.628ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
4342025-03-21T02:15:27.628ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
4352025-03-21T02:15:27.628ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
4362025-03-21T02:15:27.628ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
4372025-03-21T02:15:27.628ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
4382025-03-21T02:15:27.630ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
4392025-03-21T02:15:27.630ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "095c20ef-0e68-43f9-95b2-f421a2de01aa", "content-length": "2", "date": "Fri, 21 Mar 2025 02:15:26 GMT"} })
4402025-03-21T02:15:27.630ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
4412025-03-21T02:15:27.630ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "91378abd-6b23-4081-ae45-ff2cbdba65a3", "content-length": "52", "date": "Fri, 21 Mar 2025 02:15:26 GMT"} })
4422025-03-21T02:15:27.630ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
4432025-03-21T02:15:27.630ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
4442025-03-21T02:15:27.647ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
4452025-03-21T02:15:27.648ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "5a49b75a-5e4e-434c-a609-892bdd09b02e", "date": "Fri, 21 Mar 2025 02:15:27 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
4462025-03-21T02:15:27.648ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
4472025-03-21T02:15:27.649ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "ad545523-0d85-4e60-b72c-ca6e4feecddc", "date": "Fri, 21 Mar 2025 02:15:27 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
4482025-03-21T02:15:27.649ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
4492025-03-21T02:15:27.650ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "59eaf925-4b08-4dad-a6e3-8c65e0103268", "date": "Fri, 21 Mar 2025 02:15:27 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
4502025-03-21T02:15:27.650ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
4512025-03-21T02:15:27.652ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
4522025-03-21T02:15:27.652ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 729.159954ms iteration = 14 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
4532025-03-21T02:15:27.721ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "c402cf1c-f5c5-4b16-a94f-e76066ecee89", "date": "Fri, 21 Mar 2025 02:15:27 GMT"} })
4542025-03-21T02:15:27.721ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch = Switch0
4552025-03-21T02:15:27.721ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
4562025-03-21T02:15:27.721ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e4e48c2-d9ae-4379-81e6-9555866dfb26", "content-length": "682", "date": "Fri, 21 Mar 2025 02:15:27 GMT"} })
4572025-03-21T02:15:27.721ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
4582025-03-21T02:15:27.721ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
4592025-03-21T02:15:27.725ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
4602025-03-21T02:15:27.725ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
4612025-03-21T02:15:27.729ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
4622025-03-21T02:15:27.729ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
4632025-03-21T02:15:27.731ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
4642025-03-21T02:15:27.731ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
4652025-03-21T02:15:27.739ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 816.095287ms iteration = 23 status = {}
4662025-03-21T02:15:27.869ZDEBGnexus (producer-registration-task): registering / renewing oximeter producer lease with Nexus
4672025-03-21T02:15:27.869ZDEBGnexus (producer-registration-task): using nexus address for registration addr = [fd00:1122:3344:101::c]:12221
4682025-03-21T02:15:27.899ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::c]:12221/metrics/producers
4692025-03-21T02:15:27.900ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::c]:12221 remote_addr = [fd00:1122:3344:101::c]:42412
4702025-03-21T02:15:27.900ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::c]:42412 req_id = d9774521-d9f8-40fb-9677-58369f76e261 roles = RoleSet { roles: {} } uri = /metrics/producers
4712025-03-21T02:15:27.900ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::c]:42412 req_id = d9774521-d9f8-40fb-9677-58369f76e261 resource = Database result = Ok(()) uri = /metrics/producers
4722025-03-21T02:15:27.948ZINFOnexus: assigned collector to new producer collector_id = a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53 file = nexus/src/app/oximeter.rs:76 producer_id = e05eb799-8cf0-4702-bd3b-9994ae86118a
4732025-03-21T02:15:27.948ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 48532 local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::c]:42412 req_id = d9774521-d9f8-40fb-9677-58369f76e261 response_code = 201 uri = /metrics/producers
4742025-03-21T02:15:27.948ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "d9774521-d9f8-40fb-9677-58369f76e261", "content-length": "41", "date": "Fri, 21 Mar 2025 02:15:27 GMT"} })
4752025-03-21T02:15:27.950ZDEBGnexus (producer-registration-task): registered with nexus successfully
4762025-03-21T02:15:27.950ZDEBGnexus (producer-registration-task): pausing until time to renew lease lease_duration = 600s wait_period = 150s
4772025-03-21T02:15:29.723ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 312 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = d504d7b7-9258-40cd-9045-38c9fbf225a9 response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
4782025-03-21T02:15:36.922ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 68 reason = Timeout
4792025-03-21T02:15:36.922ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
4802025-03-21T02:15:36.922ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 68 reason = Timeout
4812025-03-21T02:15:36.922ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4822025-03-21T02:15:36.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
4832025-03-21T02:15:36.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4842025-03-21T02:15:36.925ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
4852025-03-21T02:15:36.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
4862025-03-21T02:15:36.925ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4872025-03-21T02:15:36.926ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.221033ms iteration = 68 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
4882025-03-21T02:15:36.928ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
4892025-03-21T02:15:36.928ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4902025-03-21T02:15:36.928ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4912025-03-21T02:15:36.930ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4922025-03-21T02:15:36.930ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4932025-03-21T02:15:36.932ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4942025-03-21T02:15:36.934ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
4952025-03-21T02:15:36.934ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4962025-03-21T02:15:36.934ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4972025-03-21T02:15:36.942ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
4982025-03-21T02:15:36.942ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 19.408198ms iteration = 68 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
4992025-03-21T02:15:39.723ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 384 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = 45545817-e881-4766-9045-de61ab559326 response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
5002025-03-21T02:15:46.922ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 69 reason = Timeout
5012025-03-21T02:15:46.922ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
5022025-03-21T02:15:46.922ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 69 reason = Timeout
5032025-03-21T02:15:46.922ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
5042025-03-21T02:15:46.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
5052025-03-21T02:15:46.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
5062025-03-21T02:15:46.924ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
5072025-03-21T02:15:46.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
5082025-03-21T02:15:46.925ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5092025-03-21T02:15:46.926ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.01295ms iteration = 69 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
5102025-03-21T02:15:46.927ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
5112025-03-21T02:15:46.927ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
5122025-03-21T02:15:46.927ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
5132025-03-21T02:15:46.929ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
5142025-03-21T02:15:46.930ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
5152025-03-21T02:15:46.931ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5162025-03-21T02:15:46.933ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
5172025-03-21T02:15:46.933ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
5182025-03-21T02:15:46.933ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
5192025-03-21T02:15:46.942ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
5202025-03-21T02:15:46.942ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 19.362446ms iteration = 69 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
5212025-03-21T02:15:49.723ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 416 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = 7134b42a-1d82-455a-8a7f-cbdf88ddac23 response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
5222025-03-21T02:15:54.715ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:57035 req_id = c86c8275-5fb7-4c87-b918-8a5453bfe157 roles = RoleSet { roles: {} } uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?sort_by=id_ascending
5232025-03-21T02:15:54.715ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:57035 req_id = c86c8275-5fb7-4c87-b918-8a5453bfe157 resource = Database result = Ok(()) uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?sort_by=id_ascending
5242025-03-21T02:15:54.717ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2890 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:57035 req_id = c86c8275-5fb7-4c87-b918-8a5453bfe157 response_code = 200 uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?sort_by=id_ascending
5252025-03-21T02:15:54.718ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:57035 req_id = 493b940b-8d2b-409c-b49d-690e28fbc5f2 roles = RoleSet { roles: {} } uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlMDVlYjc5OS04Y2YwLTQ3MDItYmQzYi05OTk0YWU4NjExOGEifX0%3D
5262025-03-21T02:15:54.718ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:57035 req_id = 493b940b-8d2b-409c-b49d-690e28fbc5f2 resource = Database result = Ok(()) uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlMDVlYjc5OS04Y2YwLTQ3MDItYmQzYi05OTk0YWU4NjExOGEifX0%3D
5272025-03-21T02:15:54.720ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2276 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:57035 req_id = 493b940b-8d2b-409c-b49d-690e28fbc5f2 response_code = 200 uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlMDVlYjc5OS04Y2YwLTQ3MDItYmQzYi05OTk0YWU4NjExOGEifX0%3D
5282025-03-21T02:15:56.922ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 70 reason = Timeout
5292025-03-21T02:15:56.922ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 24 reason = Timeout
5302025-03-21T02:15:56.922ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
5312025-03-21T02:15:56.922ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 24 reason = Timeout
5322025-03-21T02:15:56.922ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 70 reason = Timeout
5332025-03-21T02:15:56.922ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
5342025-03-21T02:15:56.922ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5352025-03-21T02:15:56.922ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 26 reason = Timeout
5362025-03-21T02:15:56.922ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
5372025-03-21T02:15:56.922ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 24 reason = Timeout
5382025-03-21T02:15:56.922ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 24 reason = Timeout
5392025-03-21T02:15:56.922ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
5402025-03-21T02:15:56.922ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
5412025-03-21T02:15:56.922ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 24 reason = Timeout
5422025-03-21T02:15:56.922ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
5432025-03-21T02:15:56.922ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 24 reason = Timeout
5442025-03-21T02:15:56.922ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
5452025-03-21T02:15:56.922ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 24 reason = Timeout
5462025-03-21T02:15:56.922ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
5472025-03-21T02:15:56.922ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 24 reason = Timeout
5482025-03-21T02:15:56.922ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
5492025-03-21T02:15:56.922ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
5502025-03-21T02:15:56.923ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 24 reason = Timeout
5512025-03-21T02:15:56.923ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
5522025-03-21T02:15:56.923ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
5532025-03-21T02:15:56.923ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 24 reason = Timeout
5542025-03-21T02:15:56.923ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
5552025-03-21T02:15:56.923ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 24 reason = Timeout
5562025-03-21T02:15:56.923ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
5572025-03-21T02:15:56.923ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 24 reason = Timeout
5582025-03-21T02:15:56.923ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
5592025-03-21T02:15:56.923ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 24 reason = Timeout
5602025-03-21T02:15:56.923ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
5612025-03-21T02:15:56.923ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 24 reason = Timeout
5622025-03-21T02:15:56.923ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
5632025-03-21T02:15:56.923ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 24 reason = Timeout
5642025-03-21T02:15:56.923ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
5652025-03-21T02:15:56.923ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 24 reason = Timeout
5662025-03-21T02:15:56.923ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
5672025-03-21T02:15:56.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
5682025-03-21T02:15:56.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
5692025-03-21T02:15:56.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
5702025-03-21T02:15:56.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
5712025-03-21T02:15:56.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
5722025-03-21T02:15:56.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
5732025-03-21T02:15:56.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
5742025-03-21T02:15:56.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5752025-03-21T02:15:56.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
5762025-03-21T02:15:56.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
5772025-03-21T02:15:56.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
5782025-03-21T02:15:56.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
5792025-03-21T02:15:56.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
5802025-03-21T02:15:56.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
5812025-03-21T02:15:56.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
5822025-03-21T02:15:56.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
5832025-03-21T02:15:56.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
5842025-03-21T02:15:56.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
5852025-03-21T02:15:56.925ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
5862025-03-21T02:15:56.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
5872025-03-21T02:15:56.925ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 2.977932ms iteration = 24 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
5882025-03-21T02:15:56.925ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5892025-03-21T02:15:56.926ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5902025-03-21T02:15:56.926ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5912025-03-21T02:15:56.927ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5922025-03-21T02:15:56.928ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
5932025-03-21T02:15:56.928ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
5942025-03-21T02:15:56.928ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
5952025-03-21T02:15:56.928ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
5962025-03-21T02:15:56.928ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
5972025-03-21T02:15:56.929ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
5982025-03-21T02:15:56.929ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
5992025-03-21T02:15:56.929ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6002025-03-21T02:15:56.929ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
6012025-03-21T02:15:56.929ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
6022025-03-21T02:15:56.929ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
6032025-03-21T02:15:56.929ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
6042025-03-21T02:15:56.930ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6052025-03-21T02:15:56.931ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 8.59843ms iteration = 24 status = {"errors":[],"finish_invoked_ok":[]}
6062025-03-21T02:15:56.931ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 8.64675ms iteration = 24 status = {"errors":[],"garbage_collect_requested":[]}
6072025-03-21T02:15:56.932ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6082025-03-21T02:15:56.933ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
6092025-03-21T02:15:56.933ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
6102025-03-21T02:15:56.933ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
6112025-03-21T02:15:56.934ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
6122025-03-21T02:15:56.934ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
6132025-03-21T02:15:56.934ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
6142025-03-21T02:15:56.934ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 11.688603ms iteration = 24 status = {"errors":[],"requests_created_ok":[]}
6152025-03-21T02:15:56.935ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1665, tv_nsec: 554264647 } })
6162025-03-21T02:15:56.935ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
6172025-03-21T02:15:56.935ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
6182025-03-21T02:15:56.935ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
6192025-03-21T02:15:56.936ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
6202025-03-21T02:15:56.936ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
6212025-03-21T02:15:56.936ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
6222025-03-21T02:15:56.936ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
6232025-03-21T02:15:56.936ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6242025-03-21T02:15:56.937ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 14.751897ms iteration = 70 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
6252025-03-21T02:15:56.937ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1665, tv_nsec: 556639948 } })
6262025-03-21T02:15:56.938ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6272025-03-21T02:15:56.938ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
6282025-03-21T02:15:56.938ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
6292025-03-21T02:15:56.938ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6302025-03-21T02:15:56.938ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
6312025-03-21T02:15:56.939ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
6322025-03-21T02:15:56.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
6332025-03-21T02:15:56.939ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
6342025-03-21T02:15:56.939ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
6352025-03-21T02:15:56.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
6362025-03-21T02:15:56.942ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
6372025-03-21T02:15:56.942ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6382025-03-21T02:15:56.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
6392025-03-21T02:15:56.943ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6402025-03-21T02:15:56.943ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6412025-03-21T02:15:56.945ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
6422025-03-21T02:15:56.945ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
6432025-03-21T02:15:56.945ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
6442025-03-21T02:15:56.945ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 1 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
6452025-03-21T02:15:56.945ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 23.133452ms iteration = 24 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":1,"total_instances":0,"update_sagas_queued":0}
6462025-03-21T02:15:56.946ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
6472025-03-21T02:15:56.946ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
6482025-03-21T02:15:56.946ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
6492025-03-21T02:15:56.947ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
6502025-03-21T02:15:56.947ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6512025-03-21T02:15:56.947ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6522025-03-21T02:15:56.948ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1665, tv_nsec: 567472157 } })
6532025-03-21T02:15:56.948ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
6542025-03-21T02:15:56.954ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
6552025-03-21T02:15:56.954ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 32.16063ms iteration = 70 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
6562025-03-21T02:15:56.964ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
6572025-03-21T02:15:56.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
6582025-03-21T02:15:56.976ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
6592025-03-21T02:15:56.976ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
6602025-03-21T02:15:56.977ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6612025-03-21T02:15:56.979ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
6622025-03-21T02:15:56.979ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
6632025-03-21T02:15:56.979ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
6642025-03-21T02:15:56.982ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
6652025-03-21T02:15:56.982ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 59.373284ms iteration = 24 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
6662025-03-21T02:15:56.985ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
6672025-03-21T02:15:56.986ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
6682025-03-21T02:15:56.986ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 63.604247ms iteration = 24 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
6692025-03-21T02:15:56.989ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
6702025-03-21T02:15:56.989ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
6712025-03-21T02:15:56.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
6722025-03-21T02:15:56.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
6732025-03-21T02:15:56.991ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 68.822568ms iteration = 24 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
6742025-03-21T02:15:56.991ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
6752025-03-21T02:15:56.991ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
6762025-03-21T02:15:56.991ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
6772025-03-21T02:15:56.992ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
6782025-03-21T02:15:56.992ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
6792025-03-21T02:15:56.992ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
6802025-03-21T02:15:56.993ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
6812025-03-21T02:15:56.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
6822025-03-21T02:15:56.994ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6832025-03-21T02:15:56.995ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6842025-03-21T02:15:56.997ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: ad5b28f8-6cad-4221-97f9-e84a46a6381b, lookup_type: ById(ad5b28f8-6cad-4221-97f9-e84a46a6381b) } result = Ok(())
6852025-03-21T02:15:56.997ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
6862025-03-21T02:15:56.997ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
6872025-03-21T02:15:56.998ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
6882025-03-21T02:15:56.998ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
6892025-03-21T02:15:56.998ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
6902025-03-21T02:15:56.998ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 75.982973ms iteration = 24 status = 0
6912025-03-21T02:15:56.999ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6922025-03-21T02:15:57.001ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
6932025-03-21T02:15:57.001ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
6942025-03-21T02:15:57.001ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
6952025-03-21T02:15:57.016ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
6962025-03-21T02:15:57.016ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
6972025-03-21T02:15:57.016ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "8dade4c6-f297-410b-89a8-684e5171995e", "content-length": "26", "date": "Fri, 21 Mar 2025 02:15:56 GMT"} })
6982025-03-21T02:15:57.016ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
6992025-03-21T02:15:57.016ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 93.594019ms iteration = 24 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
7002025-03-21T02:15:57.017ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
7012025-03-21T02:15:57.019ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
7022025-03-21T02:15:57.019ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
7032025-03-21T02:15:57.019ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e7bc500-91b5-459b-a225-b434aefeb38b", "content-length": "26", "date": "Fri, 21 Mar 2025 02:15:56 GMT"} })
7042025-03-21T02:15:57.019ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
7052025-03-21T02:15:57.021ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
7062025-03-21T02:15:57.031ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
7072025-03-21T02:15:57.031ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
7082025-03-21T02:15:57.031ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 108.709343ms iteration = 24 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
7092025-03-21T02:15:57.033ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 zone_address = fd00:1122:3344:101::2
7102025-03-21T02:15:57.033ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
7112025-03-21T02:15:57.033ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "704a4c2b-0e28-4366-9251-39ae46d535b2", "content-length": "26", "date": "Fri, 21 Mar 2025 02:15:56 GMT"} })
7122025-03-21T02:15:57.033ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
7132025-03-21T02:15:57.035ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
7142025-03-21T02:15:57.056ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 133.507615ms iteration = 26 status = {"physical_disks_added":0}
7152025-03-21T02:15:57.124ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
7162025-03-21T02:15:57.124ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "42fb2a1f-b9b9-42f4-914a-1843625b1911", "content-length": "2", "date": "Fri, 21 Mar 2025 02:15:56 GMT"} })
7172025-03-21T02:15:57.125ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
7182025-03-21T02:15:57.125ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
7192025-03-21T02:15:57.127ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 204.245865ms iteration = 24 status = null
7202025-03-21T02:15:57.133ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 210.176749ms iteration = 24 status = {}
7212025-03-21T02:15:57.134ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
7222025-03-21T02:15:57.134ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
7232025-03-21T02:15:57.135ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9c8a2e0-80d1-4837-88a3-362298b15da7", "content-length": "372", "date": "Fri, 21 Mar 2025 02:15:57 GMT"} })
7242025-03-21T02:15:57.135ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
7252025-03-21T02:15:57.135ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
7262025-03-21T02:15:57.160ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
7272025-03-21T02:15:57.161ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a48a159-4edf-4978-ad11-e4dff84d279b", "content-length": "691", "date": "Fri, 21 Mar 2025 02:15:57 GMT"} })
7282025-03-21T02:15:57.161ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
7292025-03-21T02:15:57.161ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
7302025-03-21T02:15:57.161ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
7312025-03-21T02:15:57.163ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 240.752402ms iteration = 24 status = {}
7322025-03-21T02:15:57.181ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
7332025-03-21T02:15:57.181ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
7342025-03-21T02:15:57.181ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "366e4bb3-cd93-4464-9cfd-18d8e1c5f45e", "date": "Fri, 21 Mar 2025 02:15:56 GMT"} })
7352025-03-21T02:15:57.181ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
7362025-03-21T02:15:57.181ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
7372025-03-21T02:15:57.183ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
7382025-03-21T02:15:57.183ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
7392025-03-21T02:15:57.186ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
7402025-03-21T02:15:57.186ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
7412025-03-21T02:15:57.189ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
7422025-03-21T02:15:57.189ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
7432025-03-21T02:15:57.189ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "93b14314-18fc-4207-93c1-4fb806c24400", "date": "Fri, 21 Mar 2025 02:15:56 GMT"} })
7442025-03-21T02:15:57.191ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 268.175919ms iteration = 24 status = {}
7452025-03-21T02:15:57.212ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
7462025-03-21T02:15:57.212ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
7472025-03-21T02:15:57.214ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 033eb27d-ce1b-4d9a-9cf5-5fe7e1ba28e0, rack_id: 9e10fa12-3248-42fd-baa3-e5de51ba7745, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(f0ee038c-25a9-431e-b4eb-d29544961ae6) } rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
7482025-03-21T02:15:57.214ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
7492025-03-21T02:15:57.215ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
7502025-03-21T02:15:57.224ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
7512025-03-21T02:15:57.225ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "1240f6ce-5b96-4350-944e-1427c4c102ee", "content-length": "134", "date": "Fri, 21 Mar 2025 02:15:56 GMT"} })
7522025-03-21T02:15:57.225ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
7532025-03-21T02:15:57.225ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = f0ee038c-25a9-431e-b4eb-d29544961ae6
7542025-03-21T02:15:57.225ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
7552025-03-21T02:15:57.225ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
7562025-03-21T02:15:57.225ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
7572025-03-21T02:15:57.225ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
7582025-03-21T02:15:57.225ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
7592025-03-21T02:15:57.225ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
7602025-03-21T02:15:57.225ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
7612025-03-21T02:15:57.225ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
7622025-03-21T02:15:57.225ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
7632025-03-21T02:15:57.225ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
7642025-03-21T02:15:57.226ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
7652025-03-21T02:15:57.227ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
7662025-03-21T02:15:57.228ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9eb273f-0c49-48fc-9072-da1bae93ebdf", "content-length": "2", "date": "Fri, 21 Mar 2025 02:15:56 GMT"} })
7672025-03-21T02:15:57.228ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
7682025-03-21T02:15:57.228ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8883a07-e3cf-48d9-a09c-add93d7aaf68", "content-length": "52", "date": "Fri, 21 Mar 2025 02:15:56 GMT"} })
7692025-03-21T02:15:57.228ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
7702025-03-21T02:15:57.228ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
7712025-03-21T02:15:57.326ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "2e38172a-1fe1-4e28-b404-532a3062ce13", "date": "Fri, 21 Mar 2025 02:15:56 GMT"} })
7722025-03-21T02:15:57.326ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch = Switch0
7732025-03-21T02:15:57.326ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
7742025-03-21T02:15:57.326ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "5bfe0839-e708-418b-b501-7a02f9906838", "content-length": "682", "date": "Fri, 21 Mar 2025 02:15:56 GMT"} })
7752025-03-21T02:15:57.326ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
7762025-03-21T02:15:57.327ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
7772025-03-21T02:15:57.330ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
7782025-03-21T02:15:57.330ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
7792025-03-21T02:15:57.333ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
7802025-03-21T02:15:57.334ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
7812025-03-21T02:15:57.336ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
7822025-03-21T02:15:57.336ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
7832025-03-21T02:15:57.342ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 419.874089ms iteration = 24 status = {}
7842025-03-21T02:15:59.723ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 290 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = 6e914e93-3559-41a0-af60-9cdadb2031ac response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
7852025-03-21T02:16:06.923ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 71 reason = Timeout
7862025-03-21T02:16:06.923ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 71 reason = Timeout
7872025-03-21T02:16:06.923ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
7882025-03-21T02:16:06.923ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7892025-03-21T02:16:06.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7902025-03-21T02:16:06.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
7912025-03-21T02:16:06.925ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7922025-03-21T02:16:06.925ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
7932025-03-21T02:16:06.926ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
7942025-03-21T02:16:06.927ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.992999ms iteration = 71 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
7952025-03-21T02:16:06.928ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
7962025-03-21T02:16:06.928ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7972025-03-21T02:16:06.928ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7982025-03-21T02:16:06.929ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7992025-03-21T02:16:06.930ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8002025-03-21T02:16:06.931ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8012025-03-21T02:16:06.933ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
8022025-03-21T02:16:06.933ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8032025-03-21T02:16:06.933ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8042025-03-21T02:16:06.941ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
8052025-03-21T02:16:06.942ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 18.251198ms iteration = 71 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
8062025-03-21T02:16:09.724ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 273 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = ae05bc8b-ee02-4754-ba33-f07d6cc26075 response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
8072025-03-21T02:16:16.923ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 72 reason = Timeout
8082025-03-21T02:16:16.923ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
8092025-03-21T02:16:16.923ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 72 reason = Timeout
8102025-03-21T02:16:16.923ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8112025-03-21T02:16:16.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
8122025-03-21T02:16:16.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8132025-03-21T02:16:16.925ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
8142025-03-21T02:16:16.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
8152025-03-21T02:16:16.926ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8162025-03-21T02:16:16.928ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
8172025-03-21T02:16:16.928ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8182025-03-21T02:16:16.928ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8192025-03-21T02:16:16.930ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8202025-03-21T02:16:16.930ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8212025-03-21T02:16:16.932ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8222025-03-21T02:16:16.934ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
8232025-03-21T02:16:16.934ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8242025-03-21T02:16:16.934ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8252025-03-21T02:16:16.941ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
8262025-03-21T02:16:16.941ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 17.514275ms iteration = 72 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
8272025-03-21T02:16:16.959ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 35.720812ms iteration = 72 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
8282025-03-21T02:16:19.723ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 296 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = a403c364-8b9e-4ed3-8139-f980465366ab response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
8292025-03-21T02:16:26.923ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 73 reason = Timeout
8302025-03-21T02:16:26.923ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
8312025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 25 reason = Timeout
8322025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 6 reason = Timeout
8332025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
8342025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
8352025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 25 reason = Timeout
8362025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 13 reason = Timeout
8372025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
8382025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
8392025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 15 reason = Timeout
8402025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 13 reason = Timeout
8412025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
8422025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 25 reason = Timeout
8432025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
8442025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 13 reason = Timeout
8452025-03-21T02:16:26.924ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-21 02:06:26.923949584 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
8462025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
8472025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 13 reason = Timeout
8482025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
8492025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
8502025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 25 reason = Timeout
8512025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
8522025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
8532025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 25 reason = Timeout
8542025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 25 reason = Timeout
8552025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
8562025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 25 reason = Timeout
8572025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
8582025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
8592025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 13 reason = Timeout
8602025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
8612025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 25 reason = Timeout
8622025-03-21T02:16:26.924ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
8632025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 25 reason = Timeout
8642025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
8652025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 13 reason = Timeout
8662025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
8672025-03-21T02:16:26.924ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
8682025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
8692025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 13 reason = Timeout
8702025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
8712025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 25 reason = Timeout
8722025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
8732025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 14 reason = Timeout
8742025-03-21T02:16:26.924ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074
8752025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 13.84\u{b5}s iteration = 14 status = {"enabled":false,"target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074"}
8762025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 13 reason = Timeout
8772025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
8782025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 27 reason = Timeout
8792025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
8802025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 15 reason = Timeout
8812025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
8822025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 25 reason = Timeout
8832025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
8842025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 25 reason = Timeout
8852025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
8862025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 25 reason = Timeout
8872025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 25 reason = Timeout
8882025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
8892025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 73 reason = Timeout
8902025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 13 reason = Timeout
8912025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8922025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 13 reason = Timeout
8932025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 25 reason = Timeout
8942025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
8952025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 25 reason = Timeout
8962025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
8972025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
8982025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
8992025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
9002025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
9012025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
9022025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
9032025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
9042025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
9052025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
9062025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
9072025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
9082025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
9092025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
9102025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
9112025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
9122025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
9132025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
9142025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
9152025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
9162025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
9172025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
9182025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
9192025-03-21T02:16:26.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
9202025-03-21T02:16:26.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
9212025-03-21T02:16:26.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
9222025-03-21T02:16:26.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
9232025-03-21T02:16:26.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9242025-03-21T02:16:26.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9252025-03-21T02:16:26.925ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
9262025-03-21T02:16:26.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
9272025-03-21T02:16:26.925ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("a1777e88-1a9c-49b1-bca1-ec6c5e1991fc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("bfbad16c-d6a4-4742-9db2-139d724c0771.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("bfbad16c-d6a4-4742-9db2-139d724c0771.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 1695, tv_nsec: 543361808 } })
9282025-03-21T02:16:26.925ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("200fb1b3-8e37-4e66-8f46-93219f5750f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("4999f8c7-3dfb-459a-ba6c-e61aa5543438.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("91dd1310-1f23-4802-b96e-b07c13e4ea67.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("91dd1310-1f23-4802-b96e-b07c13e4ea67.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:3::1))) }], valid_until: Instant { tv_sec: 1695, tv_nsec: 543387039 } })
9292025-03-21T02:16:26.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
9302025-03-21T02:16:26.925ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
9312025-03-21T02:16:26.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
9322025-03-21T02:16:26.925ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
9332025-03-21T02:16:26.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
9342025-03-21T02:16:26.925ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
9352025-03-21T02:16:26.925ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.172271ms iteration = 13 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
9362025-03-21T02:16:26.925ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } dns_group = internal
9372025-03-21T02:16:26.925ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.266342ms iteration = 13 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353","[fd00:1122:3344:3::1]:5353"]}
9382025-03-21T02:16:26.926ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
9392025-03-21T02:16:26.927ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
9402025-03-21T02:16:26.927ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9412025-03-21T02:16:26.928ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.330066ms iteration = 25 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
9422025-03-21T02:16:26.928ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-21 02:06:26.923949584 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
9432025-03-21T02:16:26.928ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 4.824984ms iteration = 13 status = {"expiration":"2025-03-21T02:06:26.923949584Z","pruned":[]}
9442025-03-21T02:16:26.928ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1695, tv_nsec: 547333967 } })
9452025-03-21T02:16:26.929ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
9462025-03-21T02:16:26.929ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
9472025-03-21T02:16:26.929ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
9482025-03-21T02:16:26.929ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
9492025-03-21T02:16:26.929ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9502025-03-21T02:16:26.930ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9512025-03-21T02:16:26.930ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
9522025-03-21T02:16:26.930ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
9532025-03-21T02:16:26.930ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
9542025-03-21T02:16:26.931ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
9552025-03-21T02:16:26.931ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
9562025-03-21T02:16:26.932ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
9572025-03-21T02:16:26.932ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
9582025-03-21T02:16:26.932ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
9592025-03-21T02:16:26.934ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 10.420962ms iteration = 25 status = {"errors":[],"finish_invoked_ok":[]}
9602025-03-21T02:16:26.934ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9612025-03-21T02:16:26.936ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 12.114191ms iteration = 13 status = {"errors":[],"found_port_ok":[]}
9622025-03-21T02:16:26.936ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
9632025-03-21T02:16:26.936ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
9642025-03-21T02:16:26.936ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9652025-03-21T02:16:26.936ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
9662025-03-21T02:16:26.936ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9672025-03-21T02:16:26.939ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
9682025-03-21T02:16:26.939ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
9692025-03-21T02:16:26.939ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
9702025-03-21T02:16:26.939ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
9712025-03-21T02:16:26.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
9722025-03-21T02:16:26.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
9732025-03-21T02:16:26.941ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
9742025-03-21T02:16:26.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
9752025-03-21T02:16:26.943ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9762025-03-21T02:16:26.944ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9772025-03-21T02:16:26.945ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 20.693701ms iteration = 25 status = {"errors":[],"garbage_collect_requested":[]}
9782025-03-21T02:16:26.945ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1695, tv_nsec: 563468978 } })
9792025-03-21T02:16:26.945ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
9802025-03-21T02:16:26.946ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
9812025-03-21T02:16:26.946ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
9822025-03-21T02:16:26.946ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
9832025-03-21T02:16:26.946ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 22.311109ms iteration = 25 status = {"errors":[],"requests_created_ok":[]}
9842025-03-21T02:16:26.948ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9852025-03-21T02:16:26.948ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
9862025-03-21T02:16:26.948ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9872025-03-21T02:16:26.948ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
9882025-03-21T02:16:26.948ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9892025-03-21T02:16:26.948ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
9902025-03-21T02:16:26.949ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
9912025-03-21T02:16:26.950ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
9922025-03-21T02:16:26.950ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
9932025-03-21T02:16:26.950ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9942025-03-21T02:16:26.951ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9952025-03-21T02:16:26.951ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
9962025-03-21T02:16:26.951ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
9972025-03-21T02:16:26.951ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
9982025-03-21T02:16:26.951ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9992025-03-21T02:16:26.952ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
10002025-03-21T02:16:26.953ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
10012025-03-21T02:16:26.953ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
10022025-03-21T02:16:26.953ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
10032025-03-21T02:16:26.954ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
10042025-03-21T02:16:26.956ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
10052025-03-21T02:16:26.957ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1695, tv_nsec: 575835794 } })
10062025-03-21T02:16:26.957ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
10072025-03-21T02:16:26.958ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10082025-03-21T02:16:26.958ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
10092025-03-21T02:16:26.958ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
10102025-03-21T02:16:26.958ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
10112025-03-21T02:16:26.958ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
10122025-03-21T02:16:26.958ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10132025-03-21T02:16:26.958ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
10142025-03-21T02:16:26.959ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
10152025-03-21T02:16:26.959ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
10162025-03-21T02:16:26.961ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
10172025-03-21T02:16:26.961ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
10182025-03-21T02:16:26.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
10192025-03-21T02:16:26.962ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
10202025-03-21T02:16:26.964ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10212025-03-21T02:16:26.965ZDEBGnexus (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 41.32078ms iteration = 6 status = {"nsuccess":5}
10222025-03-21T02:16:26.965ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10232025-03-21T02:16:26.967ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
10242025-03-21T02:16:26.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
10252025-03-21T02:16:26.968ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
10262025-03-21T02:16:26.968ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
10272025-03-21T02:16:26.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
10282025-03-21T02:16:26.969ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
10292025-03-21T02:16:26.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
10302025-03-21T02:16:26.970ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
10312025-03-21T02:16:26.970ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
10322025-03-21T02:16:26.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
10332025-03-21T02:16:26.972ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 49.038133ms iteration = 25 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
10342025-03-21T02:16:26.973ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
10352025-03-21T02:16:26.973ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
10362025-03-21T02:16:26.973ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
10372025-03-21T02:16:26.973ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
10382025-03-21T02:16:26.973ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 0481b934-0ed5-44d2-96f2-0a9adc385687 dns_zone_name = oxide.test found_so_far = 0 version = 2
10392025-03-21T02:16:26.973ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
10402025-03-21T02:16:26.974ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
10412025-03-21T02:16:26.974ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
10422025-03-21T02:16:26.975ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10432025-03-21T02:16:26.975ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10442025-03-21T02:16:26.975ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 51.688969ms iteration = 25 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
10452025-03-21T02:16:26.978ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
10462025-03-21T02:16:26.978ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
10472025-03-21T02:16:26.978ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
10482025-03-21T02:16:26.979ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
10492025-03-21T02:16:26.979ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
10502025-03-21T02:16:26.979ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
10512025-03-21T02:16:26.981ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 0481b934-0ed5-44d2-96f2-0a9adc385687 dns_zone_name = oxide.test found_so_far = 1 version = 2
10522025-03-21T02:16:26.981ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
10532025-03-21T02:16:26.981ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-21 02:04:31.662541 UTC dns_group = external generation = 2
10542025-03-21T02:16:26.981ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 57.575193ms iteration = 13 status = {"generation":2}
10552025-03-21T02:16:26.981ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
10562025-03-21T02:16:26.981ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
10572025-03-21T02:16:26.981ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
10582025-03-21T02:16:26.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
10592025-03-21T02:16:26.984ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
10602025-03-21T02:16:26.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
10612025-03-21T02:16:26.986ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10622025-03-21T02:16:26.988ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
10632025-03-21T02:16:26.988ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
10642025-03-21T02:16:26.988ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 64.558354ms iteration = 13 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
10652025-03-21T02:16:26.988ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
10662025-03-21T02:16:26.988ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
10672025-03-21T02:16:26.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
10682025-03-21T02:16:26.992ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 68.735477ms iteration = 25 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
10692025-03-21T02:16:26.994ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
10702025-03-21T02:16:26.994ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 71.049577ms iteration = 73 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
10712025-03-21T02:16:26.995ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
10722025-03-21T02:16:27.002ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10732025-03-21T02:16:27.003ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10742025-03-21T02:16:27.003ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10752025-03-21T02:16:27.005ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
10762025-03-21T02:16:27.005ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
10772025-03-21T02:16:27.005ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
10782025-03-21T02:16:27.006ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
10792025-03-21T02:16:27.006ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
10802025-03-21T02:16:27.007ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
10812025-03-21T02:16:27.007ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
10822025-03-21T02:16:27.007ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10832025-03-21T02:16:27.007ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10842025-03-21T02:16:27.010ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10852025-03-21T02:16:27.010ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10862025-03-21T02:16:27.011ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
10872025-03-21T02:16:27.011ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
10882025-03-21T02:16:27.012ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
10892025-03-21T02:16:27.012ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 88.410749ms iteration = 25 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
10902025-03-21T02:16:27.012ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10912025-03-21T02:16:27.014ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
10922025-03-21T02:16:27.015ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
10932025-03-21T02:16:27.016ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
10942025-03-21T02:16:27.016ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10952025-03-21T02:16:27.016ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10962025-03-21T02:16:27.016ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10972025-03-21T02:16:27.017ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
10982025-03-21T02:16:27.017ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
10992025-03-21T02:16:27.017ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 92.971428ms iteration = 13 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
11002025-03-21T02:16:27.019ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: ad5b28f8-6cad-4221-97f9-e84a46a6381b, lookup_type: ById(ad5b28f8-6cad-4221-97f9-e84a46a6381b) } result = Ok(())
11012025-03-21T02:16:27.019ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
11022025-03-21T02:16:27.020ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
11032025-03-21T02:16:27.021ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
11042025-03-21T02:16:27.021ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
11052025-03-21T02:16:27.021ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
11062025-03-21T02:16:27.021ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
11072025-03-21T02:16:27.021ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 97.6899ms iteration = 25 status = 0
11082025-03-21T02:16:27.022ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
11092025-03-21T02:16:27.023ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11102025-03-21T02:16:27.023ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11112025-03-21T02:16:27.023ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11122025-03-21T02:16:27.025ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11132025-03-21T02:16:27.026ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
11142025-03-21T02:16:27.026ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
11152025-03-21T02:16:27.026ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
11162025-03-21T02:16:27.027ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
11172025-03-21T02:16:27.027ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11182025-03-21T02:16:27.027ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11192025-03-21T02:16:27.027ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
11202025-03-21T02:16:27.027ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 78c1ab2b-9e58-4297-a89e-d4fa0c8a69c3 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
11212025-03-21T02:16:27.027ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
11222025-03-21T02:16:27.028ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
11232025-03-21T02:16:27.029ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11242025-03-21T02:16:27.029ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11252025-03-21T02:16:27.030ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
11262025-03-21T02:16:27.031ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 106.721568ms iteration = 73 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
11272025-03-21T02:16:27.032ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
11282025-03-21T02:16:27.032ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11292025-03-21T02:16:27.032ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11302025-03-21T02:16:27.033ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
11312025-03-21T02:16:27.033ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
11322025-03-21T02:16:27.033ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
11332025-03-21T02:16:27.033ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
11342025-03-21T02:16:27.033ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
11352025-03-21T02:16:27.035ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
11362025-03-21T02:16:27.035ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
11372025-03-21T02:16:27.035ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
11382025-03-21T02:16:27.035ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
11392025-03-21T02:16:27.035ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 78c1ab2b-9e58-4297-a89e-d4fa0c8a69c3 dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
11402025-03-21T02:16:27.035ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
11412025-03-21T02:16:27.035ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-21 02:04:30.886360 UTC dns_group = internal generation = 1
11422025-03-21T02:16:27.035ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 111.622693ms iteration = 13 status = {"generation":1}
11432025-03-21T02:16:27.038ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-21T02:04:22.853740Z, time_modified: 2025-03-21T02:04:22.853740Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 89475719-7d35-403a-9037-042f6e747a73, db_silo: Silo { identity: SiloIdentity { id: 89475719-7d35-403a-9037-042f6e747a73, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-21T02:04:31.619926Z, time_modified: 2025-03-21T02:04:31.619926Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "c6a715a41442cf931d7245f8c6684eb4210f9631513e8dd8f0b24981cd6bd960" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 89475719-7d35-403a-9037-042f6e747a73, db_silo: Silo { identity: SiloIdentity { id: 89475719-7d35-403a-9037-042f6e747a73, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-21T02:04:31.619926Z, time_modified: 2025-03-21T02:04:31.619926Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "c6a715a41442cf931d7245f8c6684eb4210f9631513e8dd8f0b24981cd6bd960" }] }) }
11442025-03-21T02:16:27.038ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 114.804299ms iteration = 13 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"89475719-7d35-403a-9037-042f6e747a73","tls_certs":["c6a715a41442cf931d7245f8c6684eb4210f9631513e8dd8f0b24981cd6bd960"]}},"default_endpoint":{"silo_id":"89475719-7d35-403a-9037-042f6e747a73","tls_certs":["c6a715a41442cf931d7245f8c6684eb4210f9631513e8dd8f0b24981cd6bd960"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
11452025-03-21T02:16:27.040ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
11462025-03-21T02:16:27.040ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
11472025-03-21T02:16:27.078ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
11482025-03-21T02:16:27.078ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
11492025-03-21T02:16:27.078ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 154.645613ms iteration = 25 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
11502025-03-21T02:16:27.086ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11512025-03-21T02:16:27.088ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
11522025-03-21T02:16:27.088ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
11532025-03-21T02:16:27.088ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
11542025-03-21T02:16:27.109ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
11552025-03-21T02:16:27.109ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 185.389408ms iteration = 13 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
11562025-03-21T02:16:27.113ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 zone_address = fd00:1122:3344:101::2
11572025-03-21T02:16:27.113ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
11582025-03-21T02:16:27.114ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "11eaa8ba-729e-456a-84fe-a72eeba1ed91", "content-length": "26", "date": "Fri, 21 Mar 2025 02:16:27 GMT"} })
11592025-03-21T02:16:27.114ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
11602025-03-21T02:16:27.115ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
11612025-03-21T02:16:27.116ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 192.111634ms iteration = 27 status = {"physical_disks_added":0}
11622025-03-21T02:16:27.118ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
11632025-03-21T02:16:27.118ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 194.571767ms iteration = 25 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
11642025-03-21T02:16:27.165ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
11652025-03-21T02:16:27.229ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
11662025-03-21T02:16:27.229ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
11672025-03-21T02:16:27.229ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "535511d6-ef74-4a05-8ffa-f730a527ac95", "content-length": "26", "date": "Fri, 21 Mar 2025 02:16:27 GMT"} })
11682025-03-21T02:16:27.229ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
11692025-03-21T02:16:27.231ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
11702025-03-21T02:16:27.248ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
11712025-03-21T02:16:27.279ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
11722025-03-21T02:16:27.285ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
11732025-03-21T02:16:27.285ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
11742025-03-21T02:16:27.286ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "4146ead5-ebf8-4baa-864c-76bb4324c302", "content-length": "26", "date": "Fri, 21 Mar 2025 02:16:27 GMT"} })
11752025-03-21T02:16:27.286ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
11762025-03-21T02:16:27.287ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
11772025-03-21T02:16:27.289ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
11782025-03-21T02:16:27.289ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
11792025-03-21T02:16:27.360ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "844038b8-3e6b-4a42-8137-3008cd2345e6", "content-length": "372", "date": "Fri, 21 Mar 2025 02:16:26 GMT"} })
11802025-03-21T02:16:27.360ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
11812025-03-21T02:16:27.360ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8a84278-aef3-424a-a737-9bfe252438b8", "content-length": "691", "date": "Fri, 21 Mar 2025 02:16:27 GMT"} })
11822025-03-21T02:16:27.360ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
11832025-03-21T02:16:27.360ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
11842025-03-21T02:16:27.360ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
11852025-03-21T02:16:27.360ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
11862025-03-21T02:16:27.361ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 437.652269ms iteration = 25 status = {}
11872025-03-21T02:16:27.376ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
11882025-03-21T02:16:27.376ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
11892025-03-21T02:16:27.376ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "f7026142-6a8a-4b46-a4bf-5e0820e34323", "date": "Fri, 21 Mar 2025 02:16:27 GMT"} })
11902025-03-21T02:16:27.376ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
11912025-03-21T02:16:27.376ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
11922025-03-21T02:16:27.379ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
11932025-03-21T02:16:27.379ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
11942025-03-21T02:16:27.379ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
11952025-03-21T02:16:27.379ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "74846cd0-527b-4fc5-9b8a-9b6f46ba85a9", "content-length": "2", "date": "Fri, 21 Mar 2025 02:16:27 GMT"} })
11962025-03-21T02:16:27.380ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
11972025-03-21T02:16:27.381ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
11982025-03-21T02:16:27.381ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
11992025-03-21T02:16:27.382ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
12002025-03-21T02:16:27.382ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
12012025-03-21T02:16:27.382ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 458.817567ms iteration = 25 status = null
12022025-03-21T02:16:27.385ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
12032025-03-21T02:16:27.385ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
12042025-03-21T02:16:27.386ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "ec2835f5-2978-4955-9962-c7dde9d2c95b", "date": "Fri, 21 Mar 2025 02:16:27 GMT"} })
12052025-03-21T02:16:27.387ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 463.383577ms iteration = 25 status = {}
12062025-03-21T02:16:27.487ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
12072025-03-21T02:16:27.488ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "42b83351-8b01-4052-86e4-32288f84e98c", "date": "Fri, 21 Mar 2025 02:16:27 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
12082025-03-21T02:16:27.488ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
12092025-03-21T02:16:27.490ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "c475ed6d-c916-4f00-a529-f208838d7bbd", "date": "Fri, 21 Mar 2025 02:16:27 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
12102025-03-21T02:16:27.490ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
12112025-03-21T02:16:27.492ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "657fe840-a32b-4084-bfca-8b6ad94b7b7c", "date": "Fri, 21 Mar 2025 02:16:27 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
12122025-03-21T02:16:27.492ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
12132025-03-21T02:16:27.494ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
12142025-03-21T02:16:27.494ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 570.310119ms iteration = 15 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
12152025-03-21T02:16:27.495ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
12162025-03-21T02:16:27.495ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "acec49e8-ec16-4095-83d2-5174906014db", "date": "Fri, 21 Mar 2025 02:16:26 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
12172025-03-21T02:16:27.495ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
12182025-03-21T02:16:27.497ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "a88c838f-8469-4003-8252-3a668ed9987a", "date": "Fri, 21 Mar 2025 02:16:26 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
12192025-03-21T02:16:27.497ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
12202025-03-21T02:16:27.499ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
12212025-03-21T02:16:27.499ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 574.843537ms iteration = 15 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
12222025-03-21T02:16:27.507ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 583.202763ms iteration = 25 status = {}
12232025-03-21T02:16:27.515ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
12242025-03-21T02:16:27.515ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
12252025-03-21T02:16:27.517ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 033eb27d-ce1b-4d9a-9cf5-5fe7e1ba28e0, rack_id: 9e10fa12-3248-42fd-baa3-e5de51ba7745, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(f0ee038c-25a9-431e-b4eb-d29544961ae6) } rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
12262025-03-21T02:16:27.517ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
12272025-03-21T02:16:27.517ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
12282025-03-21T02:16:27.527ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
12292025-03-21T02:16:27.527ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "cbb03cda-03ca-4777-82ca-97e76b7b455c", "content-length": "134", "date": "Fri, 21 Mar 2025 02:16:27 GMT"} })
12302025-03-21T02:16:27.527ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
12312025-03-21T02:16:27.527ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = f0ee038c-25a9-431e-b4eb-d29544961ae6
12322025-03-21T02:16:27.527ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
12332025-03-21T02:16:27.527ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
12342025-03-21T02:16:27.527ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
12352025-03-21T02:16:27.527ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
12362025-03-21T02:16:27.527ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
12372025-03-21T02:16:27.527ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
12382025-03-21T02:16:27.527ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
12392025-03-21T02:16:27.527ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
12402025-03-21T02:16:27.527ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
12412025-03-21T02:16:27.527ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
12422025-03-21T02:16:27.528ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
12432025-03-21T02:16:27.529ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
12442025-03-21T02:16:27.529ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e8fc191-3919-4927-acff-562fcc10f5c2", "content-length": "2", "date": "Fri, 21 Mar 2025 02:16:27 GMT"} })
12452025-03-21T02:16:27.530ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
12462025-03-21T02:16:27.530ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "29b8f44a-ed98-41bd-9334-928d3e02c079", "content-length": "52", "date": "Fri, 21 Mar 2025 02:16:27 GMT"} })
12472025-03-21T02:16:27.530ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
12482025-03-21T02:16:27.530ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
12492025-03-21T02:16:27.624ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "6e14e2f8-774b-4c3c-ac22-b28118662cc1", "date": "Fri, 21 Mar 2025 02:16:27 GMT"} })
12502025-03-21T02:16:27.624ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch = Switch0
12512025-03-21T02:16:27.624ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
12522025-03-21T02:16:27.624ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "e16a6446-a207-4bfa-be70-3a9b08a0e2e2", "content-length": "682", "date": "Fri, 21 Mar 2025 02:16:27 GMT"} })
12532025-03-21T02:16:27.624ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
12542025-03-21T02:16:27.624ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
12552025-03-21T02:16:27.628ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
12562025-03-21T02:16:27.628ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
12572025-03-21T02:16:27.632ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
12582025-03-21T02:16:27.632ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
12592025-03-21T02:16:27.633ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
12602025-03-21T02:16:27.634ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
12612025-03-21T02:16:27.642ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 718.07286ms iteration = 25 status = {}
12622025-03-21T02:16:29.724ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 267 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = 0442ffde-6ad9-41a5-9665-0f2bee05b5fd response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
12632025-03-21T02:16:36.923ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 74 reason = Timeout
12642025-03-21T02:16:36.923ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
12652025-03-21T02:16:36.923ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 74 reason = Timeout
12662025-03-21T02:16:36.923ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12672025-03-21T02:16:36.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12682025-03-21T02:16:36.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
12692025-03-21T02:16:36.925ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12702025-03-21T02:16:36.926ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
12712025-03-21T02:16:36.926ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
12722025-03-21T02:16:36.927ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.133241ms iteration = 74 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
12732025-03-21T02:16:36.928ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
12742025-03-21T02:16:36.928ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12752025-03-21T02:16:36.928ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12762025-03-21T02:16:36.930ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12772025-03-21T02:16:36.930ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12782025-03-21T02:16:36.931ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12792025-03-21T02:16:36.934ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
12802025-03-21T02:16:36.934ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12812025-03-21T02:16:36.934ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12822025-03-21T02:16:36.940ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
12832025-03-21T02:16:36.940ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 17.092388ms iteration = 74 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
12842025-03-21T02:16:39.715ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::c]:12221 remote_addr = [fd00:1122:3344:101::d]:35844
12852025-03-21T02:16:39.715ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:35844 req_id = 6103d8a4-975b-4a17-85ea-7da69ff02e13 roles = RoleSet { roles: {} } uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?sort_by=id_ascending
12862025-03-21T02:16:39.715ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:35844 req_id = 6103d8a4-975b-4a17-85ea-7da69ff02e13 resource = Database result = Ok(()) uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?sort_by=id_ascending
12872025-03-21T02:16:39.718ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3287 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:35844 req_id = 6103d8a4-975b-4a17-85ea-7da69ff02e13 response_code = 200 uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?sort_by=id_ascending
12882025-03-21T02:16:39.718ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:35844 req_id = aa87a22f-45be-49c4-a119-724bbfb521bc roles = RoleSet { roles: {} } uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlMDVlYjc5OS04Y2YwLTQ3MDItYmQzYi05OTk0YWU4NjExOGEifX0%3D
12892025-03-21T02:16:39.719ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:35844 req_id = aa87a22f-45be-49c4-a119-724bbfb521bc resource = Database result = Ok(()) uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlMDVlYjc5OS04Y2YwLTQ3MDItYmQzYi05OTk0YWU4NjExOGEifX0%3D
12902025-03-21T02:16:39.721ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2285 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:35844 req_id = aa87a22f-45be-49c4-a119-724bbfb521bc response_code = 200 uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlMDVlYjc5OS04Y2YwLTQ3MDItYmQzYi05OTk0YWU4NjExOGEifX0%3D
12912025-03-21T02:16:39.724ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 253 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = 6ce9e185-4046-4667-8843-aed80ff7f22e response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
12922025-03-21T02:16:46.924ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 75 reason = Timeout
12932025-03-21T02:16:46.924ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 75 reason = Timeout
12942025-03-21T02:16:46.924ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
12952025-03-21T02:16:46.924ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12962025-03-21T02:16:46.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
12972025-03-21T02:16:46.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12982025-03-21T02:16:46.926ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12992025-03-21T02:16:46.929ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
13002025-03-21T02:16:46.929ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
13012025-03-21T02:16:46.929ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
13022025-03-21T02:16:46.931ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
13032025-03-21T02:16:46.931ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
13042025-03-21T02:16:46.933ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13052025-03-21T02:16:46.935ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
13062025-03-21T02:16:46.935ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
13072025-03-21T02:16:46.935ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
13082025-03-21T02:16:46.945ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
13092025-03-21T02:16:46.945ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.85497ms iteration = 75 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
13102025-03-21T02:16:46.955ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
13112025-03-21T02:16:46.955ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
13122025-03-21T02:16:46.986ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 62.477228ms iteration = 75 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
13132025-03-21T02:16:49.724ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 268 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = 66b38ee5-404b-4c53-93af-53423f630926 response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
13142025-03-21T02:16:56.923ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 76 reason = Timeout
13152025-03-21T02:16:56.923ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 26 reason = Timeout
13162025-03-21T02:16:56.923ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
13172025-03-21T02:16:56.923ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 28 reason = Timeout
13182025-03-21T02:16:56.923ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
13192025-03-21T02:16:56.923ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 26 reason = Timeout
13202025-03-21T02:16:56.923ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 26 reason = Timeout
13212025-03-21T02:16:56.923ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
13222025-03-21T02:16:56.923ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 26 reason = Timeout
13232025-03-21T02:16:56.923ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
13242025-03-21T02:16:56.923ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
13252025-03-21T02:16:56.923ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 26 reason = Timeout
13262025-03-21T02:16:56.923ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
13272025-03-21T02:16:56.923ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
13282025-03-21T02:16:56.923ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 76 reason = Timeout
13292025-03-21T02:16:56.923ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
13302025-03-21T02:16:56.923ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 26 reason = Timeout
13312025-03-21T02:16:56.923ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 26 reason = Timeout
13322025-03-21T02:16:56.923ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
13332025-03-21T02:16:56.923ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 26 reason = Timeout
13342025-03-21T02:16:56.923ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
13352025-03-21T02:16:56.923ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
13362025-03-21T02:16:56.923ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 26 reason = Timeout
13372025-03-21T02:16:56.923ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
13382025-03-21T02:16:56.923ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 26 reason = Timeout
13392025-03-21T02:16:56.923ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
13402025-03-21T02:16:56.923ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 26 reason = Timeout
13412025-03-21T02:16:56.923ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
13422025-03-21T02:16:56.923ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 26 reason = Timeout
13432025-03-21T02:16:56.923ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
13442025-03-21T02:16:56.923ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 26 reason = Timeout
13452025-03-21T02:16:56.923ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 26 reason = Timeout
13462025-03-21T02:16:56.923ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
13472025-03-21T02:16:56.923ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
13482025-03-21T02:16:56.923ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
13492025-03-21T02:16:56.923ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 26 reason = Timeout
13502025-03-21T02:16:56.923ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
13512025-03-21T02:16:56.923ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 26 reason = Timeout
13522025-03-21T02:16:56.923ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
13532025-03-21T02:16:56.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
13542025-03-21T02:16:56.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
13552025-03-21T02:16:56.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
13562025-03-21T02:16:56.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
13572025-03-21T02:16:56.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
13582025-03-21T02:16:56.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
13592025-03-21T02:16:56.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
13602025-03-21T02:16:56.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
13612025-03-21T02:16:56.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
13622025-03-21T02:16:56.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
13632025-03-21T02:16:56.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
13642025-03-21T02:16:56.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
13652025-03-21T02:16:56.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
13662025-03-21T02:16:56.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
13672025-03-21T02:16:56.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
13682025-03-21T02:16:56.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
13692025-03-21T02:16:56.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
13702025-03-21T02:16:56.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
13712025-03-21T02:16:56.925ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13722025-03-21T02:16:56.926ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 2.467343ms iteration = 26 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
13732025-03-21T02:16:56.926ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13742025-03-21T02:16:56.927ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13752025-03-21T02:16:56.927ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
13762025-03-21T02:16:56.928ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
13772025-03-21T02:16:56.928ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
13782025-03-21T02:16:56.928ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
13792025-03-21T02:16:56.928ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
13802025-03-21T02:16:56.928ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
13812025-03-21T02:16:56.929ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13822025-03-21T02:16:56.930ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
13832025-03-21T02:16:56.930ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
13842025-03-21T02:16:56.930ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
13852025-03-21T02:16:56.931ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
13862025-03-21T02:16:56.931ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
13872025-03-21T02:16:56.931ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
13882025-03-21T02:16:56.932ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1725, tv_nsec: 550821621 } })
13892025-03-21T02:16:56.933ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
13902025-03-21T02:16:56.933ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
13912025-03-21T02:16:56.933ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
13922025-03-21T02:16:56.934ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 10.506763ms iteration = 26 status = {"errors":[],"finish_invoked_ok":[]}
13932025-03-21T02:16:56.935ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1725, tv_nsec: 553191823 } })
13942025-03-21T02:16:56.935ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13952025-03-21T02:16:56.935ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
13962025-03-21T02:16:56.935ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
13972025-03-21T02:16:56.935ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13982025-03-21T02:16:56.935ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
13992025-03-21T02:16:56.937ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
14002025-03-21T02:16:56.937ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
14012025-03-21T02:16:56.937ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
14022025-03-21T02:16:56.937ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
14032025-03-21T02:16:56.938ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
14042025-03-21T02:16:56.938ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
14052025-03-21T02:16:56.938ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
14062025-03-21T02:16:56.938ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
14072025-03-21T02:16:56.941ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
14082025-03-21T02:16:56.941ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1725, tv_nsec: 559647645 } })
14092025-03-21T02:16:56.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
14102025-03-21T02:16:56.941ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
14112025-03-21T02:16:56.942ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
14122025-03-21T02:16:56.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
14132025-03-21T02:16:56.943ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14142025-03-21T02:16:56.946ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
14152025-03-21T02:16:56.946ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
14162025-03-21T02:16:56.946ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
14172025-03-21T02:16:56.969ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
14182025-03-21T02:16:56.969ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 45.544303ms iteration = 76 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
14192025-03-21T02:16:56.974ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
14202025-03-21T02:16:56.975ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
14212025-03-21T02:16:56.980ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 57.381259ms iteration = 76 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
14222025-03-21T02:16:56.981ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
14232025-03-21T02:16:56.981ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
14242025-03-21T02:16:56.981ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 58.162193ms iteration = 26 status = {"errors":[],"requests_created_ok":[]}
14252025-03-21T02:16:56.983ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 60.033825ms iteration = 26 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
14262025-03-21T02:16:56.986ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14272025-03-21T02:16:56.986ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
14282025-03-21T02:16:56.986ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
14292025-03-21T02:16:56.987ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
14302025-03-21T02:16:56.987ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
14312025-03-21T02:16:56.987ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
14322025-03-21T02:16:56.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
14332025-03-21T02:16:56.988ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
14342025-03-21T02:16:56.988ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
14352025-03-21T02:16:56.988ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
14362025-03-21T02:16:56.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
14372025-03-21T02:16:56.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
14382025-03-21T02:16:56.989ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14392025-03-21T02:16:56.989ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14402025-03-21T02:16:56.990ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
14412025-03-21T02:16:56.990ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
14422025-03-21T02:16:56.990ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
14432025-03-21T02:16:56.990ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
14442025-03-21T02:16:56.991ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
14452025-03-21T02:16:56.991ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
14462025-03-21T02:16:56.991ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
14472025-03-21T02:16:56.991ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
14482025-03-21T02:16:56.992ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
14492025-03-21T02:16:56.992ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
14502025-03-21T02:16:56.992ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14512025-03-21T02:16:56.992ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14522025-03-21T02:16:56.993ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 69.754824ms iteration = 26 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
14532025-03-21T02:16:56.993ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 69.955708ms iteration = 26 status = {"errors":[],"garbage_collect_requested":[]}
14542025-03-21T02:16:56.994ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
14552025-03-21T02:16:56.994ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
14562025-03-21T02:16:56.994ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
14572025-03-21T02:16:56.994ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 71.171709ms iteration = 26 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
14582025-03-21T02:16:56.995ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14592025-03-21T02:16:56.998ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: ad5b28f8-6cad-4221-97f9-e84a46a6381b, lookup_type: ById(ad5b28f8-6cad-4221-97f9-e84a46a6381b) } result = Ok(())
14602025-03-21T02:16:56.998ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
14612025-03-21T02:16:56.998ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
14622025-03-21T02:16:56.998ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 75.010326ms iteration = 28 status = {"physical_disks_added":0}
14632025-03-21T02:16:56.999ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 76.263958ms iteration = 26 status = 0
14642025-03-21T02:16:57.017ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
14652025-03-21T02:16:57.017ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
14662025-03-21T02:16:57.018ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "c581a968-971a-4b9f-b72e-01de9cabc4c0", "content-length": "26", "date": "Fri, 21 Mar 2025 02:16:56 GMT"} })
14672025-03-21T02:16:57.018ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
14682025-03-21T02:16:57.019ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
14692025-03-21T02:16:57.020ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 96.745064ms iteration = 26 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
14702025-03-21T02:16:57.022ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
14712025-03-21T02:16:57.022ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
14722025-03-21T02:16:57.023ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
14732025-03-21T02:16:57.023ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
14742025-03-21T02:16:57.023ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 99.93011ms iteration = 26 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
14752025-03-21T02:16:57.023ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14762025-03-21T02:16:57.026ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
14772025-03-21T02:16:57.026ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
14782025-03-21T02:16:57.026ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
14792025-03-21T02:16:57.026ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
14802025-03-21T02:16:57.027ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "faeed511-401a-45c6-bc34-3b945991edbe", "content-length": "26", "date": "Fri, 21 Mar 2025 02:16:57 GMT"} })
14812025-03-21T02:16:57.027ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
14822025-03-21T02:16:57.027ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
14832025-03-21T02:16:57.029ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
14842025-03-21T02:16:57.033ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 zone_address = fd00:1122:3344:101::2
14852025-03-21T02:16:57.033ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
14862025-03-21T02:16:57.033ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1503ea0-59c0-4744-a8a1-c984ae3e6c26", "content-length": "26", "date": "Fri, 21 Mar 2025 02:16:57 GMT"} })
14872025-03-21T02:16:57.033ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
14882025-03-21T02:16:57.035ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
14892025-03-21T02:16:57.057ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
14902025-03-21T02:16:57.058ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 134.192176ms iteration = 26 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
14912025-03-21T02:16:57.080ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
14922025-03-21T02:16:57.080ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
14932025-03-21T02:16:57.081ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "994e8ad8-b6fe-4922-9294-5573d4c43c5a", "content-length": "372", "date": "Fri, 21 Mar 2025 02:16:57 GMT"} })
14942025-03-21T02:16:57.081ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14952025-03-21T02:16:57.081ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14962025-03-21T02:16:57.091ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
14972025-03-21T02:16:57.091ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
14982025-03-21T02:16:57.092ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14992025-03-21T02:16:57.094ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
15002025-03-21T02:16:57.094ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
15012025-03-21T02:16:57.094ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
15022025-03-21T02:16:57.096ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15032025-03-21T02:16:57.098ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
15042025-03-21T02:16:57.098ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
15052025-03-21T02:16:57.098ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
15062025-03-21T02:16:57.111ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
15072025-03-21T02:16:57.112ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "65ac2e87-9025-4d29-922a-4a42df55b307", "content-length": "2", "date": "Fri, 21 Mar 2025 02:16:56 GMT"} })
15082025-03-21T02:16:57.114ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
15092025-03-21T02:16:57.114ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
15102025-03-21T02:16:57.116ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 192.790966ms iteration = 26 status = null
15112025-03-21T02:16:57.121ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 197.955656ms iteration = 26 status = {}
15122025-03-21T02:16:57.127ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
15132025-03-21T02:16:57.127ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
15142025-03-21T02:16:57.129ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "b814fe3e-1ba0-4093-b84f-16f94309389a", "date": "Fri, 21 Mar 2025 02:16:57 GMT"} })
15152025-03-21T02:16:57.129ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
15162025-03-21T02:16:57.130ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
15172025-03-21T02:16:57.132ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
15182025-03-21T02:16:57.132ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
15192025-03-21T02:16:57.135ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
15202025-03-21T02:16:57.135ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
15212025-03-21T02:16:57.138ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
15222025-03-21T02:16:57.138ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
15232025-03-21T02:16:57.142ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "2ba6cbd7-8369-4566-909c-1d3126c6bcb0", "date": "Fri, 21 Mar 2025 02:16:57 GMT"} })
15242025-03-21T02:16:57.143ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 220.319626ms iteration = 26 status = {}
15252025-03-21T02:16:57.208ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
15262025-03-21T02:16:57.208ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
15272025-03-21T02:16:57.217ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
15282025-03-21T02:16:57.217ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f5f016a-fdbd-48f8-8200-2a0776ca358c", "content-length": "691", "date": "Fri, 21 Mar 2025 02:16:56 GMT"} })
15292025-03-21T02:16:57.218ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
15302025-03-21T02:16:57.218ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
15312025-03-21T02:16:57.218ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
15322025-03-21T02:16:57.219ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 033eb27d-ce1b-4d9a-9cf5-5fe7e1ba28e0, rack_id: 9e10fa12-3248-42fd-baa3-e5de51ba7745, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(f0ee038c-25a9-431e-b4eb-d29544961ae6) } rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
15332025-03-21T02:16:57.219ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
15342025-03-21T02:16:57.219ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
15352025-03-21T02:16:57.219ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 296.37791ms iteration = 26 status = {}
15362025-03-21T02:16:57.229ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
15372025-03-21T02:16:57.229ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "2367e335-3a92-4046-b399-3f97af60fb70", "content-length": "134", "date": "Fri, 21 Mar 2025 02:16:56 GMT"} })
15382025-03-21T02:16:57.229ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
15392025-03-21T02:16:57.229ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = f0ee038c-25a9-431e-b4eb-d29544961ae6
15402025-03-21T02:16:57.229ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
15412025-03-21T02:16:57.229ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
15422025-03-21T02:16:57.229ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
15432025-03-21T02:16:57.229ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
15442025-03-21T02:16:57.229ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
15452025-03-21T02:16:57.229ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
15462025-03-21T02:16:57.230ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
15472025-03-21T02:16:57.230ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
15482025-03-21T02:16:57.230ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
15492025-03-21T02:16:57.230ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
15502025-03-21T02:16:57.230ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
15512025-03-21T02:16:57.232ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
15522025-03-21T02:16:57.232ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "8331b0a0-1217-472e-b322-a317f8d893cc", "content-length": "2", "date": "Fri, 21 Mar 2025 02:16:56 GMT"} })
15532025-03-21T02:16:57.232ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
15542025-03-21T02:16:57.232ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff76870c-a71d-45c2-b54e-b683cf16765d", "content-length": "52", "date": "Fri, 21 Mar 2025 02:16:56 GMT"} })
15552025-03-21T02:16:57.232ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
15562025-03-21T02:16:57.232ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
15572025-03-21T02:16:57.327ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "2c48e250-819e-48f1-bb36-9f4e3b8432c7", "date": "Fri, 21 Mar 2025 02:16:56 GMT"} })
15582025-03-21T02:16:57.327ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch = Switch0
15592025-03-21T02:16:57.327ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
15602025-03-21T02:16:57.327ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd9bbe4a-8ee6-47f7-9afe-edf3aea5e5c3", "content-length": "682", "date": "Fri, 21 Mar 2025 02:16:56 GMT"} })
15612025-03-21T02:16:57.327ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
15622025-03-21T02:16:57.327ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
15632025-03-21T02:16:57.330ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
15642025-03-21T02:16:57.330ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
15652025-03-21T02:16:57.334ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
15662025-03-21T02:16:57.334ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
15672025-03-21T02:16:57.336ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
15682025-03-21T02:16:57.336ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
15692025-03-21T02:16:57.343ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 420.137674ms iteration = 26 status = {}
15702025-03-21T02:16:59.725ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 261 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = b1397beb-b7cc-495a-81d3-0bee19feaff1 response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
15712025-03-21T02:17:06.924ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 77 reason = Timeout
15722025-03-21T02:17:06.924ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15732025-03-21T02:17:06.924ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 77 reason = Timeout
15742025-03-21T02:17:06.924ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
15752025-03-21T02:17:06.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15762025-03-21T02:17:06.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
15772025-03-21T02:17:06.926ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15782025-03-21T02:17:06.927ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
15792025-03-21T02:17:06.927ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
15802025-03-21T02:17:06.928ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
15812025-03-21T02:17:06.928ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15822025-03-21T02:17:06.929ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15832025-03-21T02:17:06.929ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.950416ms iteration = 77 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
15842025-03-21T02:17:06.930ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15852025-03-21T02:17:06.930ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15862025-03-21T02:17:06.932ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15872025-03-21T02:17:06.934ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
15882025-03-21T02:17:06.934ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15892025-03-21T02:17:06.934ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15902025-03-21T02:17:06.941ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
15912025-03-21T02:17:06.941ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 16.801903ms iteration = 77 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
15922025-03-21T02:17:09.715ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:35844 req_id = f62dc0fa-39b9-40fa-b83c-9f1996dd0db0 roles = RoleSet { roles: {} } uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?sort_by=id_ascending
15932025-03-21T02:17:09.715ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:35844 req_id = f62dc0fa-39b9-40fa-b83c-9f1996dd0db0 resource = Database result = Ok(()) uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?sort_by=id_ascending
15942025-03-21T02:17:09.718ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 3425 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:35844 req_id = f62dc0fa-39b9-40fa-b83c-9f1996dd0db0 response_code = 200 uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?sort_by=id_ascending
15952025-03-21T02:17:09.719ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:35844 req_id = b5528bf9-ee82-4bdf-aa1a-018320f84f72 roles = RoleSet { roles: {} } uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlMDVlYjc5OS04Y2YwLTQ3MDItYmQzYi05OTk0YWU4NjExOGEifX0%3D
15962025-03-21T02:17:09.719ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:35844 req_id = b5528bf9-ee82-4bdf-aa1a-018320f84f72 resource = Database result = Ok(()) uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlMDVlYjc5OS04Y2YwLTQ3MDItYmQzYi05OTk0YWU4NjExOGEifX0%3D
15972025-03-21T02:17:09.722ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 2871 local_addr = [fd00:1122:3344:101::c]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:35844 req_id = b5528bf9-ee82-4bdf-aa1a-018320f84f72 response_code = 200 uri = /metrics/collectors/a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJlMDVlYjc5OS04Y2YwLTQ3MDItYmQzYi05OTk0YWU4NjExOGEifX0%3D
15982025-03-21T02:17:09.725ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 271 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = 5e555dda-b195-4e44-bedb-f8f3d1ce791c response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
15992025-03-21T02:17:16.924ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 78 reason = Timeout
16002025-03-21T02:17:16.924ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
16012025-03-21T02:17:16.924ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 78 reason = Timeout
16022025-03-21T02:17:16.924ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16032025-03-21T02:17:16.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
16042025-03-21T02:17:16.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
16052025-03-21T02:17:16.926ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
16062025-03-21T02:17:16.926ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
16072025-03-21T02:17:16.927ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16082025-03-21T02:17:16.928ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.690184ms iteration = 78 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
16092025-03-21T02:17:16.929ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
16102025-03-21T02:17:16.929ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16112025-03-21T02:17:16.929ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
16122025-03-21T02:17:16.931ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16132025-03-21T02:17:16.931ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
16142025-03-21T02:17:16.933ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16152025-03-21T02:17:16.935ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
16162025-03-21T02:17:16.935ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16172025-03-21T02:17:16.935ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
16182025-03-21T02:17:16.942ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
16192025-03-21T02:17:16.942ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 17.601666ms iteration = 78 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
16202025-03-21T02:17:19.724ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 299 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = baf6c291-a94c-45e9-be64-4940a30d34cf response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
16212025-03-21T02:17:26.924ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 79 reason = Timeout
16222025-03-21T02:17:26.924ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 27 reason = Timeout
16232025-03-21T02:17:26.924ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
16242025-03-21T02:17:26.924ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
16252025-03-21T02:17:26.924ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 15 reason = Timeout
16262025-03-21T02:17:26.924ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:80 reason = blueprint disabled target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074
16272025-03-21T02:17:26.924ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 14 reason = Timeout
16282025-03-21T02:17:26.924ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 20.37\u{b5}s iteration = 15 status = {"enabled":false,"target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074"}
16292025-03-21T02:17:26.924ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 14 reason = Timeout
16302025-03-21T02:17:26.924ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 27 reason = Timeout
16312025-03-21T02:17:26.924ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
16322025-03-21T02:17:26.924ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
16332025-03-21T02:17:26.924ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
16342025-03-21T02:17:26.924ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
16352025-03-21T02:17:26.924ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 14 reason = Timeout
16362025-03-21T02:17:26.924ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
16372025-03-21T02:17:26.924ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 27 reason = Timeout
16382025-03-21T02:17:26.924ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
16392025-03-21T02:17:26.924ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 27 reason = Timeout
16402025-03-21T02:17:26.924ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
16412025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 27 reason = Timeout
16422025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
16432025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 14 reason = Timeout
16442025-03-21T02:17:26.925ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:56
16452025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 27 reason = Timeout
16462025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
16472025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
16482025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 16 reason = Timeout
16492025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 14 reason = Timeout
16502025-03-21T02:17:26.925ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-03-21 02:07:26.924925155 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
16512025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 14 reason = Timeout
16522025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
16532025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
16542025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 79 reason = Timeout
16552025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 27 reason = Timeout
16562025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16572025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
16582025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 27 reason = Timeout
16592025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
16602025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 27 reason = Timeout
16612025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 27 reason = Timeout
16622025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 27 reason = Timeout
16632025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
16642025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 14 reason = Timeout
16652025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
16662025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 27 reason = Timeout
16672025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
16682025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
16692025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 16 reason = Timeout
16702025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 27 reason = Timeout
16712025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
16722025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 27 reason = Timeout
16732025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
16742025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 14 reason = Timeout
16752025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 29 reason = Timeout
16762025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
16772025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 27 reason = Timeout
16782025-03-21T02:17:26.925ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
16792025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 14 reason = Timeout
16802025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
16812025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
16822025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
16832025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 27 reason = Timeout
16842025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
16852025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 14 reason = Timeout
16862025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
16872025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
16882025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
16892025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
16902025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
16912025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
16922025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
16932025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
16942025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
16952025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
16962025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
16972025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
16982025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
16992025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17002025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
17012025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
17022025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
17032025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
17042025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
17052025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
17062025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
17072025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
17082025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
17092025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
17102025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
17112025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
17122025-03-21T02:17:26.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
17132025-03-21T02:17:26.925ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("a1777e88-1a9c-49b1-bca1-ec6c5e1991fc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("bfbad16c-d6a4-4742-9db2-139d724c0771.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("bfbad16c-d6a4-4742-9db2-139d724c0771.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::8))) }], valid_until: Instant { tv_sec: 1755, tv_nsec: 543549537 } })
17142025-03-21T02:17:26.925ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("200fb1b3-8e37-4e66-8f46-93219f5750f7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("4999f8c7-3dfb-459a-ba6c-e61aa5543438.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("91dd1310-1f23-4802-b96e-b07c13e4ea67.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("91dd1310-1f23-4802-b96e-b07c13e4ea67.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:3::1))) }], valid_until: Instant { tv_sec: 1755, tv_nsec: 543559678 } })
17152025-03-21T02:17:26.926ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
17162025-03-21T02:17:26.926ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.295113ms iteration = 14 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
17172025-03-21T02:17:26.926ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } dns_group = internal
17182025-03-21T02:17:26.926ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.317692ms iteration = 14 status = {"addresses":["[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:1::1]:5353","[fd00:1122:3344:3::1]:5353"]}
17192025-03-21T02:17:26.927ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
17202025-03-21T02:17:26.927ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
17212025-03-21T02:17:26.928ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1755, tv_nsec: 546509819 } })
17222025-03-21T02:17:26.929ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
17232025-03-21T02:17:26.929ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
17242025-03-21T02:17:26.929ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.294235ms iteration = 27 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
17252025-03-21T02:17:26.929ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
17262025-03-21T02:17:26.930ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1755, tv_nsec: 547909553 } })
17272025-03-21T02:17:26.930ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
17282025-03-21T02:17:26.932ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17292025-03-21T02:17:26.932ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17302025-03-21T02:17:26.933ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-03-21 02:07:26.924925155 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
17312025-03-21T02:17:26.933ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 9.003077ms iteration = 14 status = {"expiration":"2025-03-21T02:07:26.924925155Z","pruned":[]}
17322025-03-21T02:17:26.933ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17332025-03-21T02:17:26.934ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
17342025-03-21T02:17:26.934ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
17352025-03-21T02:17:26.935ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
17362025-03-21T02:17:26.935ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
17372025-03-21T02:17:26.935ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
17382025-03-21T02:17:26.935ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 10.679906ms iteration = 27 status = {"errors":[],"requests_created_ok":[]}
17392025-03-21T02:17:26.935ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
17402025-03-21T02:17:26.937ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
17412025-03-21T02:17:26.937ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
17422025-03-21T02:17:26.937ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17432025-03-21T02:17:26.937ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17442025-03-21T02:17:26.939ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17452025-03-21T02:17:26.940ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17462025-03-21T02:17:26.940ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
17472025-03-21T02:17:26.940ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
17482025-03-21T02:17:26.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
17492025-03-21T02:17:26.941ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 16.990266ms iteration = 14 status = {"errors":[],"found_port_ok":[]}
17502025-03-21T02:17:26.943ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
17512025-03-21T02:17:26.943ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
17522025-03-21T02:17:26.943ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
17532025-03-21T02:17:26.943ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
17542025-03-21T02:17:26.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
17552025-03-21T02:17:26.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
17562025-03-21T02:17:26.945ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
17572025-03-21T02:17:26.945ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
17582025-03-21T02:17:26.945ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
17592025-03-21T02:17:26.945ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
17602025-03-21T02:17:26.945ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
17612025-03-21T02:17:26.945ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
17622025-03-21T02:17:26.947ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17632025-03-21T02:17:26.948ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17642025-03-21T02:17:26.949ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
17652025-03-21T02:17:26.949ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
17662025-03-21T02:17:26.949ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
17672025-03-21T02:17:26.951ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
17682025-03-21T02:17:26.951ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
17692025-03-21T02:17:26.951ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 26.495982ms iteration = 79 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
17702025-03-21T02:17:26.951ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
17712025-03-21T02:17:26.952ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
17722025-03-21T02:17:26.952ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
17732025-03-21T02:17:26.953ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
17742025-03-21T02:17:26.953ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17752025-03-21T02:17:26.954ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17762025-03-21T02:17:26.954ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
17772025-03-21T02:17:26.954ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
17782025-03-21T02:17:26.954ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17792025-03-21T02:17:26.955ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
17802025-03-21T02:17:26.955ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
17812025-03-21T02:17:26.955ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
17822025-03-21T02:17:26.956ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
17832025-03-21T02:17:26.957ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
17842025-03-21T02:17:26.957ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
17852025-03-21T02:17:26.957ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
17862025-03-21T02:17:26.958ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
17872025-03-21T02:17:26.960ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
17882025-03-21T02:17:26.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
17892025-03-21T02:17:26.963ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17902025-03-21T02:17:26.966ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
17912025-03-21T02:17:26.966ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
17922025-03-21T02:17:26.966ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17932025-03-21T02:17:26.969ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
17942025-03-21T02:17:26.969ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
17952025-03-21T02:17:26.969ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 44.073748ms iteration = 27 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
17962025-03-21T02:17:26.974ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 49.677075ms iteration = 27 status = {"errors":[],"garbage_collect_requested":[]}
17972025-03-21T02:17:26.976ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17982025-03-21T02:17:26.978ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
17992025-03-21T02:17:26.978ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
18002025-03-21T02:17:26.978ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
18012025-03-21T02:17:26.981ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 56.359171ms iteration = 27 status = {"errors":[],"finish_invoked_ok":[]}
18022025-03-21T02:17:26.981ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
18032025-03-21T02:17:26.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
18042025-03-21T02:17:26.983ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
18052025-03-21T02:17:26.983ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
18062025-03-21T02:17:26.984ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
18072025-03-21T02:17:26.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
18082025-03-21T02:17:26.985ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 60.738718ms iteration = 27 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
18092025-03-21T02:17:26.986ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 61.44908ms iteration = 27 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
18102025-03-21T02:17:26.987ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
18112025-03-21T02:17:26.987ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 62.942155ms iteration = 79 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
18122025-03-21T02:17:26.988ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
18132025-03-21T02:17:26.988ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:167 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
18142025-03-21T02:17:26.988ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 63.100009ms iteration = 14 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
18152025-03-21T02:17:26.988ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 63.700198ms iteration = 29 status = {"physical_disks_added":0}
18162025-03-21T02:17:26.991ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
18172025-03-21T02:17:26.991ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
18182025-03-21T02:17:26.992ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
18192025-03-21T02:17:26.993ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
18202025-03-21T02:17:26.994ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18212025-03-21T02:17:26.994ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18222025-03-21T02:17:26.995ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18232025-03-21T02:17:26.995ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
18242025-03-21T02:17:26.995ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
18252025-03-21T02:17:26.995ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
18262025-03-21T02:17:26.996ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
18272025-03-21T02:17:26.996ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
18282025-03-21T02:17:26.997ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
18292025-03-21T02:17:26.997ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
18302025-03-21T02:17:26.997ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
18312025-03-21T02:17:26.997ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
18322025-03-21T02:17:26.997ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
18332025-03-21T02:17:26.998ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
18342025-03-21T02:17:26.998ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: ad5b28f8-6cad-4221-97f9-e84a46a6381b, lookup_type: ById(ad5b28f8-6cad-4221-97f9-e84a46a6381b) } result = Ok(())
18352025-03-21T02:17:26.998ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
18362025-03-21T02:17:26.998ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
18372025-03-21T02:17:26.998ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
18382025-03-21T02:17:26.998ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
18392025-03-21T02:17:26.998ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:140 found = 0 not_ready_to_be_deleted = 0
18402025-03-21T02:17:26.998ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 73.671933ms iteration = 14 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
18412025-03-21T02:17:26.998ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
18422025-03-21T02:17:26.999ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18432025-03-21T02:17:27.001ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18442025-03-21T02:17:27.001ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 76.805547ms iteration = 27 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
18452025-03-21T02:17:27.003ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
18462025-03-21T02:17:27.003ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
18472025-03-21T02:17:27.003ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
18482025-03-21T02:17:27.003ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
18492025-03-21T02:17:27.003ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
18502025-03-21T02:17:27.004ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
18512025-03-21T02:17:27.006ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1755, tv_nsec: 624571498 } })
18522025-03-21T02:17:27.007ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
18532025-03-21T02:17:27.007ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18542025-03-21T02:17:27.008ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
18552025-03-21T02:17:27.008ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 83.320711ms iteration = 27 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
18562025-03-21T02:17:27.008ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 83.461193ms iteration = 27 status = 0
18572025-03-21T02:17:27.009ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
18582025-03-21T02:17:27.009ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
18592025-03-21T02:17:27.010ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
18602025-03-21T02:17:27.011ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
18612025-03-21T02:17:27.011ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
18622025-03-21T02:17:27.013ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18632025-03-21T02:17:27.015ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
18642025-03-21T02:17:27.015ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
18652025-03-21T02:17:27.016ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
18662025-03-21T02:17:27.016ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
18672025-03-21T02:17:27.017ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
18682025-03-21T02:17:27.020ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
18692025-03-21T02:17:27.031ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
18702025-03-21T02:17:27.048ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
18712025-03-21T02:17:27.048ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
18722025-03-21T02:17:27.048ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
18732025-03-21T02:17:27.048ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
18742025-03-21T02:17:27.048ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
18752025-03-21T02:17:27.050ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18762025-03-21T02:17:27.050ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
18772025-03-21T02:17:27.050ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 125.622197ms iteration = 27 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
18782025-03-21T02:17:27.052ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
18792025-03-21T02:17:27.052ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
18802025-03-21T02:17:27.052ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
18812025-03-21T02:17:27.053ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
18822025-03-21T02:17:27.053ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 0481b934-0ed5-44d2-96f2-0a9adc385687 dns_zone_name = oxide.test found_so_far = 0 version = 2
18832025-03-21T02:17:27.053ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
18842025-03-21T02:17:27.053ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
18852025-03-21T02:17:27.054ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18862025-03-21T02:17:27.055ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
18872025-03-21T02:17:27.055ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 78c1ab2b-9e58-4297-a89e-d4fa0c8a69c3 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
18882025-03-21T02:17:27.055ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
18892025-03-21T02:17:27.055ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
18902025-03-21T02:17:27.057ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18912025-03-21T02:17:27.057ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
18922025-03-21T02:17:27.057ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
18932025-03-21T02:17:27.057ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
18942025-03-21T02:17:27.059ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
18952025-03-21T02:17:27.059ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
18962025-03-21T02:17:27.059ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 0481b934-0ed5-44d2-96f2-0a9adc385687 dns_zone_name = oxide.test found_so_far = 1 version = 2
18972025-03-21T02:17:27.059ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
18982025-03-21T02:17:27.059ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-03-21 02:04:31.662541 UTC dns_group = external generation = 2
18992025-03-21T02:17:27.059ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 134.4424ms iteration = 14 status = {"generation":2}
19002025-03-21T02:17:27.059ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
19012025-03-21T02:17:27.063ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
19022025-03-21T02:17:27.063ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
19032025-03-21T02:17:27.063ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ec5ef0a-902e-40f1-a641-228d265124da", "content-length": "26", "date": "Fri, 21 Mar 2025 02:17:27 GMT"} })
19042025-03-21T02:17:27.063ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
19052025-03-21T02:17:27.063ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-03-21T02:04:22.853740Z, time_modified: 2025-03-21T02:04:22.853740Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 89475719-7d35-403a-9037-042f6e747a73, db_silo: Silo { identity: SiloIdentity { id: 89475719-7d35-403a-9037-042f6e747a73, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-21T02:04:31.619926Z, time_modified: 2025-03-21T02:04:31.619926Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "c6a715a41442cf931d7245f8c6684eb4210f9631513e8dd8f0b24981cd6bd960" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 89475719-7d35-403a-9037-042f6e747a73, db_silo: Silo { identity: SiloIdentity { id: 89475719-7d35-403a-9037-042f6e747a73, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-03-21T02:04:31.619926Z, time_modified: 2025-03-21T02:04:31.619926Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "c6a715a41442cf931d7245f8c6684eb4210f9631513e8dd8f0b24981cd6bd960" }] }) }
19062025-03-21T02:17:27.064ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 139.205123ms iteration = 14 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"89475719-7d35-403a-9037-042f6e747a73","tls_certs":["c6a715a41442cf931d7245f8c6684eb4210f9631513e8dd8f0b24981cd6bd960"]}},"default_endpoint":{"silo_id":"89475719-7d35-403a-9037-042f6e747a73","tls_certs":["c6a715a41442cf931d7245f8c6684eb4210f9631513e8dd8f0b24981cd6bd960"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
19072025-03-21T02:17:27.065ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
19082025-03-21T02:17:27.066ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 78c1ab2b-9e58-4297-a89e-d4fa0c8a69c3 dns_zone_name = control-plane.oxide.internal found_so_far = 52 version = 1
19092025-03-21T02:17:27.066ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
19102025-03-21T02:17:27.066ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-03-21 02:04:30.886360 UTC dns_group = internal generation = 1
19112025-03-21T02:17:27.066ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 141.456412ms iteration = 14 status = {"generation":1}
19122025-03-21T02:17:27.067ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
19132025-03-21T02:17:27.067ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
19142025-03-21T02:17:27.068ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "c98152fd-b8f0-4b22-9d8d-2e381a9de416", "content-length": "26", "date": "Fri, 21 Mar 2025 02:17:27 GMT"} })
19152025-03-21T02:17:27.068ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
19162025-03-21T02:17:27.070ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
19172025-03-21T02:17:27.092ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
19182025-03-21T02:17:27.092ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
19192025-03-21T02:17:27.099ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
19202025-03-21T02:17:27.099ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
19212025-03-21T02:17:27.101ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19222025-03-21T02:17:27.103ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
19232025-03-21T02:17:27.103ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
19242025-03-21T02:17:27.103ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
19252025-03-21T02:17:27.105ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19262025-03-21T02:17:27.106ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
19272025-03-21T02:17:27.106ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
19282025-03-21T02:17:27.106ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
19292025-03-21T02:17:27.138ZWARNnexus (BackgroundTasks): instance reincarnation completed with errors background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:103 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0 query_errors = 0 restart_errors = 0
19302025-03-21T02:17:27.138ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 213.465817ms iteration = 14 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
19312025-03-21T02:17:27.168ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 zone_address = fd00:1122:3344:101::2
19322025-03-21T02:17:27.168ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
19332025-03-21T02:17:27.169ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0ef67cc-a1a3-49ae-a8ab-21abaf55a413", "content-length": "26", "date": "Fri, 21 Mar 2025 02:17:27 GMT"} })
19342025-03-21T02:17:27.169ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
19352025-03-21T02:17:27.174ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
19362025-03-21T02:17:27.189ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
19372025-03-21T02:17:27.189ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3d61fb7-aa03-4c7d-a231-5a634bb1a8ef", "content-length": "2", "date": "Fri, 21 Mar 2025 02:17:26 GMT"} })
19382025-03-21T02:17:27.191ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
19392025-03-21T02:17:27.192ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
19402025-03-21T02:17:27.192ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
19412025-03-21T02:17:27.193ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "48a2bc7a-2324-472a-8ad1-37a910b79e8a", "date": "Fri, 21 Mar 2025 02:17:26 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
19422025-03-21T02:17:27.193ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
19432025-03-21T02:17:27.193ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 268.81725ms iteration = 27 status = null
19442025-03-21T02:17:27.194ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "d3add939-b564-4154-9e8b-91a596af3b8c", "date": "Fri, 21 Mar 2025 02:17:26 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
19452025-03-21T02:17:27.194ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
19462025-03-21T02:17:27.196ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
19472025-03-21T02:17:27.196ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 271.029208ms iteration = 16 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
19482025-03-21T02:17:27.197ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
19492025-03-21T02:17:27.218ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 293.793254ms iteration = 27 status = {}
19502025-03-21T02:17:27.223ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
19512025-03-21T02:17:27.223ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
19522025-03-21T02:17:27.245ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
19532025-03-21T02:17:27.261ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "842a26a7-b75a-4d85-a06e-9087999cbef1", "content-length": "372", "date": "Fri, 21 Mar 2025 02:17:27 GMT"} })
19542025-03-21T02:17:27.261ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
19552025-03-21T02:17:27.261ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4f182b2-0aa9-4bba-be66-d1faa8120585", "content-length": "691", "date": "Fri, 21 Mar 2025 02:17:27 GMT"} })
19562025-03-21T02:17:27.261ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
19572025-03-21T02:17:27.261ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
19582025-03-21T02:17:27.261ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
19592025-03-21T02:17:27.261ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
19602025-03-21T02:17:27.262ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 337.595597ms iteration = 27 status = {}
19612025-03-21T02:17:27.271ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
19622025-03-21T02:17:27.272ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "9f58cd3b-b2f5-45d9-81ad-337f4bab0d49", "date": "Fri, 21 Mar 2025 02:17:27 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
19632025-03-21T02:17:27.272ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
19642025-03-21T02:17:27.273ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "5c2dc81e-4ca3-4ace-99fc-e4d0447da815", "date": "Fri, 21 Mar 2025 02:17:27 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
19652025-03-21T02:17:27.273ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
19662025-03-21T02:17:27.275ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "cb3a8f98-8da8-47b1-adb0-eec0c4ac301e", "date": "Fri, 21 Mar 2025 02:17:27 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
19672025-03-21T02:17:27.275ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
19682025-03-21T02:17:27.276ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:1::1]:5353, [fd00:1122:3344:3::1]:5353] }
19692025-03-21T02:17:27.276ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 351.897796ms iteration = 16 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
19702025-03-21T02:17:27.312ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
19712025-03-21T02:17:27.312ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
19722025-03-21T02:17:27.312ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "a4ae0763-4284-4b93-8bcd-ea3c0f811c32", "date": "Fri, 21 Mar 2025 02:17:27 GMT"} })
19732025-03-21T02:17:27.312ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
19742025-03-21T02:17:27.312ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
19752025-03-21T02:17:27.315ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
19762025-03-21T02:17:27.315ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
19772025-03-21T02:17:27.319ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
19782025-03-21T02:17:27.319ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
19792025-03-21T02:17:27.323ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
19802025-03-21T02:17:27.323ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
19812025-03-21T02:17:27.323ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "a29f41b0-d33f-4476-8537-8a90d1447420", "date": "Fri, 21 Mar 2025 02:17:27 GMT"} })
19822025-03-21T02:17:27.325ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 400.409471ms iteration = 27 status = {}
19832025-03-21T02:17:27.331ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
19842025-03-21T02:17:27.332ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
19852025-03-21T02:17:27.333ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 033eb27d-ce1b-4d9a-9cf5-5fe7e1ba28e0, rack_id: 9e10fa12-3248-42fd-baa3-e5de51ba7745, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(f0ee038c-25a9-431e-b4eb-d29544961ae6) } rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
19862025-03-21T02:17:27.333ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
19872025-03-21T02:17:27.334ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
19882025-03-21T02:17:27.343ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
19892025-03-21T02:17:27.344ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ef65a14-69da-4710-ab6b-da0a60365b50", "content-length": "134", "date": "Fri, 21 Mar 2025 02:17:26 GMT"} })
19902025-03-21T02:17:27.344ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
19912025-03-21T02:17:27.344ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = f0ee038c-25a9-431e-b4eb-d29544961ae6
19922025-03-21T02:17:27.344ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
19932025-03-21T02:17:27.344ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
19942025-03-21T02:17:27.344ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
19952025-03-21T02:17:27.344ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
19962025-03-21T02:17:27.344ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
19972025-03-21T02:17:27.344ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
19982025-03-21T02:17:27.344ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
19992025-03-21T02:17:27.344ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
20002025-03-21T02:17:27.344ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
20012025-03-21T02:17:27.345ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
20022025-03-21T02:17:27.345ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
20032025-03-21T02:17:27.346ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
20042025-03-21T02:17:27.346ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "776ca6a0-169d-4b81-927a-8635248fddae", "content-length": "2", "date": "Fri, 21 Mar 2025 02:17:26 GMT"} })
20052025-03-21T02:17:27.346ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
20062025-03-21T02:17:27.346ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "e41239ce-ed3b-4f76-bbcd-ce49ef10fb4c", "content-length": "52", "date": "Fri, 21 Mar 2025 02:17:26 GMT"} })
20072025-03-21T02:17:27.346ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
20082025-03-21T02:17:27.346ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
20092025-03-21T02:17:27.444ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "7ef1bcf3-76aa-41e4-ba87-189f703b486d", "date": "Fri, 21 Mar 2025 02:17:27 GMT"} })
20102025-03-21T02:17:27.444ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch = Switch0
20112025-03-21T02:17:27.444ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
20122025-03-21T02:17:27.445ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "335bb4b1-4406-4e21-ba65-b4c96d3eab81", "content-length": "682", "date": "Fri, 21 Mar 2025 02:17:27 GMT"} })
20132025-03-21T02:17:27.445ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
20142025-03-21T02:17:27.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
20152025-03-21T02:17:27.448ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
20162025-03-21T02:17:27.448ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
20172025-03-21T02:17:27.451ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
20182025-03-21T02:17:27.452ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
20192025-03-21T02:17:27.453ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
20202025-03-21T02:17:27.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
20212025-03-21T02:17:27.460ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 534.918803ms iteration = 27 status = {}
20222025-03-21T02:17:29.724ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 270 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = d1361395-39b2-49eb-86d5-cbd69be795b9 response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
20232025-03-21T02:17:36.924ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 80 reason = Timeout
20242025-03-21T02:17:36.924ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
20252025-03-21T02:17:36.924ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 80 reason = Timeout
20262025-03-21T02:17:36.924ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
20272025-03-21T02:17:36.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
20282025-03-21T02:17:36.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
20292025-03-21T02:17:36.926ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
20302025-03-21T02:17:36.926ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20312025-03-21T02:17:36.926ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
20322025-03-21T02:17:36.928ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
20332025-03-21T02:17:36.928ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
20342025-03-21T02:17:36.928ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
20352025-03-21T02:17:36.930ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
20362025-03-21T02:17:36.930ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
20372025-03-21T02:17:36.931ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20382025-03-21T02:17:36.933ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
20392025-03-21T02:17:36.933ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
20402025-03-21T02:17:36.934ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
20412025-03-21T02:17:36.940ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
20422025-03-21T02:17:36.940ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 16.690791ms iteration = 80 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
20432025-03-21T02:17:36.956ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 32.258912ms iteration = 80 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
20442025-03-21T02:17:39.725ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 296 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = 8cf4a35b-e409-46f7-8acc-05bd184da830 response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
20452025-03-21T02:17:46.924ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 81 reason = Timeout
20462025-03-21T02:17:46.924ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
20472025-03-21T02:17:46.924ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 81 reason = Timeout
20482025-03-21T02:17:46.924ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
20492025-03-21T02:17:46.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
20502025-03-21T02:17:46.924ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
20512025-03-21T02:17:46.925ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
20522025-03-21T02:17:46.926ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
20532025-03-21T02:17:46.926ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20542025-03-21T02:17:46.927ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.527831ms iteration = 81 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
20552025-03-21T02:17:46.929ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
20562025-03-21T02:17:46.929ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
20572025-03-21T02:17:46.929ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
20582025-03-21T02:17:46.930ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
20592025-03-21T02:17:46.930ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
20602025-03-21T02:17:46.932ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20612025-03-21T02:17:46.935ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
20622025-03-21T02:17:46.935ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
20632025-03-21T02:17:46.935ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
20642025-03-21T02:17:46.943ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
20652025-03-21T02:17:46.943ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 19.104003ms iteration = 81 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
20662025-03-21T02:17:49.724ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 298 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = 01b3f68e-d47d-43b2-9e49-30a85fc47c4d response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a
20672025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 82 reason = Timeout
20682025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
20692025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 28 reason = Timeout
20702025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 28 reason = Timeout
20712025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 28 reason = Timeout
20722025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 82 reason = Timeout
20732025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
20742025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
20752025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 28 reason = Timeout
20762025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
20772025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 28 reason = Timeout
20782025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
20792025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
20802025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 28 reason = Timeout
20812025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
20822025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 28 reason = Timeout
20832025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 28 reason = Timeout
20842025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
20852025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
20862025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 28 reason = Timeout
20872025-03-21T02:17:56.925ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
20882025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
20892025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 28 reason = Timeout
20902025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
20912025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 28 reason = Timeout
20922025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
20932025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 28 reason = Timeout
20942025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
20952025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 28 reason = Timeout
20962025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
20972025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 28 reason = Timeout
20982025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
20992025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 28 reason = Timeout
21002025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
21012025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
21022025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 30 reason = Timeout
21032025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
21042025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 28 reason = Timeout
21052025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
21062025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
21072025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
21082025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21092025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
21102025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
21112025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
21122025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
21132025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
21142025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
21152025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
21162025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
21172025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
21182025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
21192025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
21202025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
21212025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
21222025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
21232025-03-21T02:17:56.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
21242025-03-21T02:17:56.927ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
21252025-03-21T02:17:56.927ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
21262025-03-21T02:17:56.928ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21272025-03-21T02:17:56.928ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21282025-03-21T02:17:56.929ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.363966ms iteration = 28 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
21292025-03-21T02:17:56.930ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1785, tv_nsec: 547773353 } })
21302025-03-21T02:17:56.930ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1171
21312025-03-21T02:17:56.931ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
21322025-03-21T02:17:56.931ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21332025-03-21T02:17:56.931ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21342025-03-21T02:17:56.931ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
21352025-03-21T02:17:56.931ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
21362025-03-21T02:17:56.932ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
21372025-03-21T02:17:56.933ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1785, tv_nsec: 551115581 } })
21382025-03-21T02:17:56.934ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1171
21392025-03-21T02:17:56.934ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21402025-03-21T02:17:56.935ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21412025-03-21T02:17:56.936ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
21422025-03-21T02:17:56.936ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
21432025-03-21T02:17:56.936ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
21442025-03-21T02:17:56.936ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
21452025-03-21T02:17:56.936ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
21462025-03-21T02:17:56.936ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
21472025-03-21T02:17:56.936ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
21482025-03-21T02:17:56.938ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
21492025-03-21T02:17:56.938ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
21502025-03-21T02:17:56.938ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
21512025-03-21T02:17:56.940ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21522025-03-21T02:17:56.942ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21532025-03-21T02:17:56.942ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
21542025-03-21T02:17:56.942ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
21552025-03-21T02:17:56.942ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
21562025-03-21T02:17:56.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21572025-03-21T02:17:56.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
21582025-03-21T02:17:56.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
21592025-03-21T02:17:56.944ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
21602025-03-21T02:17:56.945ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
21612025-03-21T02:17:56.949ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-ad5b28f8-6cad-4221-97f9-e84a46a6381b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::2))) }], valid_until: Instant { tv_sec: 1785, tv_nsec: 566381917 } })
21622025-03-21T02:17:56.949ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1171 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
21632025-03-21T02:17:56.976ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 51.330794ms iteration = 28 status = {"errors":[],"finish_invoked_ok":[]}
21642025-03-21T02:17:56.978ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
21652025-03-21T02:17:56.979ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
21662025-03-21T02:17:56.980ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 55.245392ms iteration = 28 status = {"errors":[],"garbage_collect_requested":[]}
21672025-03-21T02:17:56.982ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
21682025-03-21T02:17:56.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
21692025-03-21T02:17:56.985ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
21702025-03-21T02:17:56.985ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21712025-03-21T02:17:56.985ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
21722025-03-21T02:17:56.987ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21732025-03-21T02:17:56.988ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 63.082709ms iteration = 82 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
21742025-03-21T02:17:56.988ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21752025-03-21T02:17:56.989ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
21762025-03-21T02:17:56.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
21772025-03-21T02:17:56.990ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21782025-03-21T02:17:56.990ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
21792025-03-21T02:17:56.990ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
21802025-03-21T02:17:56.990ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
21812025-03-21T02:17:56.990ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: e1d039bd-ccd9-4583-8c3c-337d8242f074, lookup_type: ById(e1d039bd-ccd9-4583-8c3c-337d8242f074) } result = Ok(())
21822025-03-21T02:17:56.990ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21832025-03-21T02:17:56.990ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
21842025-03-21T02:17:56.990ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 65.648093ms iteration = 28 status = {"errors":[],"requests_created_ok":[]}
21852025-03-21T02:17:56.990ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21862025-03-21T02:17:56.991ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
21872025-03-21T02:17:56.991ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
21882025-03-21T02:17:56.992ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
21892025-03-21T02:17:56.992ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
21902025-03-21T02:17:56.992ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: ad5b28f8-6cad-4221-97f9-e84a46a6381b, lookup_type: ById(ad5b28f8-6cad-4221-97f9-e84a46a6381b) } result = Ok(())
21912025-03-21T02:17:56.992ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
21922025-03-21T02:17:56.992ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
21932025-03-21T02:17:56.992ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
21942025-03-21T02:17:56.992ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
21952025-03-21T02:17:56.992ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
21962025-03-21T02:17:56.993ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
21972025-03-21T02:17:56.993ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
21982025-03-21T02:17:56.993ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
21992025-03-21T02:17:56.993ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 68.681885ms iteration = 28 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
22002025-03-21T02:17:56.995ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22012025-03-21T02:17:56.996ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 70.949055ms iteration = 28 status = 0
22022025-03-21T02:17:56.996ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
22032025-03-21T02:17:56.996ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
22042025-03-21T02:17:56.997ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
22052025-03-21T02:17:56.997ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
22062025-03-21T02:17:56.997ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
22072025-03-21T02:17:56.998ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 72.901009ms iteration = 28 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
22082025-03-21T02:17:56.999ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:555 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
22092025-03-21T02:17:56.999ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 74.542138ms iteration = 28 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
22102025-03-21T02:17:56.999ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22112025-03-21T02:17:57.001ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 75.788529ms iteration = 30 status = {"physical_disks_added":0}
22122025-03-21T02:17:57.001ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
22132025-03-21T02:17:57.001ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
22142025-03-21T02:17:57.001ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
22152025-03-21T02:17:57.001ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 original_time_created = 2025-03-21 02:04:20.112247 UTC target_id = e1d039bd-ccd9-4583-8c3c-337d8242f074 time_created = 2025-03-21 02:04:20.112247 UTC
22162025-03-21T02:17:57.001ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 76.690475ms iteration = 82 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"e1d039bd-ccd9-4583-8c3c-337d8242f074","time_created":"2025-03-21T02:04:20.112247Z"}
22172025-03-21T02:17:57.001ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
22182025-03-21T02:17:57.001ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
22192025-03-21T02:17:57.002ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0adc769-678b-4278-a5da-e3e7dba3ddc8", "content-length": "26", "date": "Fri, 21 Mar 2025 02:17:56 GMT"} })
22202025-03-21T02:17:57.002ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
22212025-03-21T02:17:57.004ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
22222025-03-21T02:17:57.016ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 90.77979ms iteration = 28 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
22232025-03-21T02:17:57.019ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1179 zone_address = fd00:1122:3344:101::2
22242025-03-21T02:17:57.019ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
22252025-03-21T02:17:57.019ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "5889ba02-eaca-482d-a91d-013d0fcccabd", "content-length": "26", "date": "Fri, 21 Mar 2025 02:17:56 GMT"} })
22262025-03-21T02:17:57.019ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1184 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
22272025-03-21T02:17:57.021ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
22282025-03-21T02:17:57.021ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1179 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 zone_address = fd00:1122:3344:101::2
22292025-03-21T02:17:57.021ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
22302025-03-21T02:17:57.021ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "a10e17dc-44cf-450f-bc50-faced2e4d0c4", "content-length": "26", "date": "Fri, 21 Mar 2025 02:17:56 GMT"} })
22312025-03-21T02:17:57.021ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1184 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
22322025-03-21T02:17:57.023ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1219 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
22332025-03-21T02:17:57.028ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
22342025-03-21T02:17:57.028ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
22352025-03-21T02:17:57.030ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 104.388488ms iteration = 28 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
22362025-03-21T02:17:57.035ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
22372025-03-21T02:17:57.035ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
22382025-03-21T02:17:57.037ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22392025-03-21T02:17:57.039ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
22402025-03-21T02:17:57.039ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
22412025-03-21T02:17:57.039ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
22422025-03-21T02:17:57.072ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:360
22432025-03-21T02:17:57.072ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 146.872937ms iteration = 28 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
22442025-03-21T02:17:57.075ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 150.194175ms iteration = 28 status = {}
22452025-03-21T02:17:57.104ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
22462025-03-21T02:17:57.105ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6f40d6d-4edc-4a69-a58c-02c151472f2f", "content-length": "2", "date": "Fri, 21 Mar 2025 02:17:56 GMT"} })
22472025-03-21T02:17:57.106ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
22482025-03-21T02:17:57.106ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
22492025-03-21T02:17:57.108ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 183.783809ms iteration = 28 status = null
22502025-03-21T02:17:57.129ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
22512025-03-21T02:17:57.129ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
22522025-03-21T02:17:57.132ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
22532025-03-21T02:17:57.132ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
22542025-03-21T02:17:57.134ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22552025-03-21T02:17:57.135ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
22562025-03-21T02:17:57.135ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
22572025-03-21T02:17:57.136ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
22582025-03-21T02:17:57.137ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22592025-03-21T02:17:57.139ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
22602025-03-21T02:17:57.139ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
22612025-03-21T02:17:57.139ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
22622025-03-21T02:17:57.180ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
22632025-03-21T02:17:57.180ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
22642025-03-21T02:17:57.181ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:151 port = SwitchPort { id: 033eb27d-ce1b-4d9a-9cf5-5fe7e1ba28e0, rack_id: 9e10fa12-3248-42fd-baa3-e5de51ba7745, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(f0ee038c-25a9-431e-b4eb-d29544961ae6) } rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
22652025-03-21T02:17:57.182ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
22662025-03-21T02:17:57.182ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
22672025-03-21T02:17:57.191ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
22682025-03-21T02:17:57.191ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b0a7cfd-91d9-4952-9dc3-fcddb65db86d", "content-length": "134", "date": "Fri, 21 Mar 2025 02:17:56 GMT"} })
22692025-03-21T02:17:57.191ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1746 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = "qsfp0"
22702025-03-21T02:17:57.191ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1773 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = f0ee038c-25a9-431e-b4eb-d29544961ae6
22712025-03-21T02:17:57.191ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:383 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
22722025-03-21T02:17:57.191ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:387 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
22732025-03-21T02:17:57.191ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:396 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
22742025-03-21T02:17:57.191ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
22752025-03-21T02:17:57.191ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:407 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
22762025-03-21T02:17:57.191ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1920 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
22772025-03-21T02:17:57.192ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:413 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
22782025-03-21T02:17:57.192ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1956 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.1.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
22792025-03-21T02:17:57.192ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:422 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745
22802025-03-21T02:17:57.192ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
22812025-03-21T02:17:57.193ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
22822025-03-21T02:17:57.194ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
22832025-03-21T02:17:57.195ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv4", status: 200, headers: {"content-type": "application/json", "x-request-id": "98d86949-ae08-4d3a-a906-d305898d2908", "content-length": "2", "date": "Fri, 21 Mar 2025 02:17:56 GMT"} })
22842025-03-21T02:17:57.195ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
22852025-03-21T02:17:57.195ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12224/loopback/ipv6", status: 200, headers: {"content-type": "application/json", "x-request-id": "740f0879-8722-4fe9-8245-56e8cb6353a9", "content-length": "52", "date": "Fri, 21 Mar 2025 02:17:56 GMT"} })
22862025-03-21T02:17:57.195ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:471 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch_location = Switch0
22872025-03-21T02:17:57.195ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
22882025-03-21T02:17:57.203ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "fbd10231-1b79-4ffb-bf44-9148ab7ca7ce", "content-length": "372", "date": "Fri, 21 Mar 2025 02:17:57 GMT"} })
22892025-03-21T02:17:57.203ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
22902025-03-21T02:17:57.203ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
22912025-03-21T02:17:57.237ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
22922025-03-21T02:17:57.237ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "e90bb191-ab93-4d04-8634-1684fa47d8ce", "content-length": "691", "date": "Fri, 21 Mar 2025 02:17:57 GMT"} })
22932025-03-21T02:17:57.238ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = buskin
22942025-03-21T02:17:57.238ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = buskin
22952025-03-21T02:17:57.238ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = buskin
22962025-03-21T02:17:57.239ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 313.862954ms iteration = 28 status = {}
22972025-03-21T02:17:57.245ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = buskin
22982025-03-21T02:17:57.245ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
22992025-03-21T02:17:57.246ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "933057a9-fe46-441f-8c6e-b559601f5cb7", "date": "Fri, 21 Mar 2025 02:17:57 GMT"} })
23002025-03-21T02:17:57.246ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
23012025-03-21T02:17:57.246ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
23022025-03-21T02:17:57.248ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
23032025-03-21T02:17:57.248ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
23042025-03-21T02:17:57.251ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
23052025-03-21T02:17:57.251ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
23062025-03-21T02:17:57.254ZINFOnexus (BackgroundTasks): VPC route manager sled ad5b28f8-6cad-4221-97f9-e84a46a6381b push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
23072025-03-21T02:17:57.254ZDEBGnexus (BackgroundTasks): client request SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
23082025-03-21T02:17:57.257ZDEBGnexus (BackgroundTasks): client response SledAgent = ad5b28f8-6cad-4221-97f9-e84a46a6381b background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "5e30fc27-8f42-4b08-b7a4-f1c3b01d9ce3", "date": "Fri, 21 Mar 2025 02:17:57 GMT"} })
23092025-03-21T02:17:57.258ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 333.71463ms iteration = 28 status = {}
23102025-03-21T02:17:57.300ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "5c08b1b7-d4a9-44b5-a21a-048d43c307e5", "date": "Fri, 21 Mar 2025 02:17:57 GMT"} })
23112025-03-21T02:17:57.301ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:774 rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 switch = Switch0
23122025-03-21T02:17:57.301ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
23132025-03-21T02:17:57.301ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 9e10fa12-3248-42fd-baa3-e5de51ba7745 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "25764d80-9e0e-48dc-96de-0ba04a9e2eda", "content-length": "682", "date": "Fri, 21 Mar 2025 02:17:57 GMT"} })
23142025-03-21T02:17:57.301ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
23152025-03-21T02:17:57.301ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
23162025-03-21T02:17:57.305ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
23172025-03-21T02:17:57.305ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
23182025-03-21T02:17:57.308ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
23192025-03-21T02:17:57.308ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
23202025-03-21T02:17:57.310ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
23212025-03-21T02:17:57.310ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
23222025-03-21T02:17:57.317ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 392.772738ms iteration = 28 status = {}
23232025-03-21T02:17:57.953ZDEBGnexus (producer-registration-task): registering / renewing oximeter producer lease with Nexus
23242025-03-21T02:17:57.953ZDEBGnexus (producer-registration-task): using nexus address for registration addr = [fd00:1122:3344:101::c]:12221
23252025-03-21T02:17:57.984ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::c]:12221/metrics/producers
23262025-03-21T02:17:57.984ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = [fd00:1122:3344:101::c]:12221 remote_addr = [fd00:1122:3344:101::c]:35968
23272025-03-21T02:17:57.984ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::c]:35968 req_id = e68fb967-cf33-4611-b62a-345c042fa4f8 roles = RoleSet { roles: {} } uri = /metrics/producers
23282025-03-21T02:17:57.984ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::c]:35968 req_id = e68fb967-cf33-4611-b62a-345c042fa4f8 resource = Database result = Ok(()) uri = /metrics/producers
23292025-03-21T02:17:58.019ZINFOnexus: assigned collector to new producer collector_id = a7d3a3cd-88e0-4c1e-8497-fc0c78e2ad53 file = nexus/src/app/oximeter.rs:76 producer_id = e05eb799-8cf0-4702-bd3b-9994ae86118a
23302025-03-21T02:17:58.019ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 35665 local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::c]:35968 req_id = e68fb967-cf33-4611-b62a-345c042fa4f8 response_code = 201 uri = /metrics/producers
23312025-03-21T02:17:58.020ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "e68fb967-cf33-4611-b62a-345c042fa4f8", "content-length": "41", "date": "Fri, 21 Mar 2025 02:17:57 GMT"} })
23322025-03-21T02:17:58.022ZDEBGnexus (producer-registration-task): registered with nexus successfully
23332025-03-21T02:17:58.022ZDEBGnexus (producer-registration-task): pausing until time to renew lease lease_duration = 600s wait_period = 150s
23342025-03-21T02:17:59.725ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 322 local_addr = [fd00:1122:3344:101::c]:42506 method = GET remote_addr = [fd00:1122:3344:101::d]:64804 req_id = d9dbe3d7-cdf3-4280-91e8-b656085e19e6 response_code = 200 uri = /e05eb799-8cf0-4702-bd3b-9994ae86118a