Max level shown:
12025-07-17T11:30:01.692ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 344 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = de8ba329-1f36-46ae-b105-6a9818638797 response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
22025-07-17T11:30:04.763ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 52 reason = Timeout
32025-07-17T11:30:04.763ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
42025-07-17T11:30:04.763ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
52025-07-17T11:30:04.765ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62025-07-17T11:30:04.768ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
72025-07-17T11:30:04.768ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
82025-07-17T11:30:04.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
92025-07-17T11:30:04.771ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
102025-07-17T11:30:04.771ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 7.827984ms iteration = 52 status = {"chicken_switches_updated":false}
112025-07-17T11:30:09.763ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 27 reason = Timeout
122025-07-17T11:30:09.763ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 27 reason = Timeout
132025-07-17T11:30:09.763ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
142025-07-17T11:30:09.763ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 53 reason = Timeout
152025-07-17T11:30:09.763ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
162025-07-17T11:30:09.763ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
172025-07-17T11:30:09.763ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
182025-07-17T11:30:09.763ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
192025-07-17T11:30:09.763ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
202025-07-17T11:30:09.765ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
212025-07-17T11:30:09.766ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
222025-07-17T11:30:09.766ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
232025-07-17T11:30:09.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
242025-07-17T11:30:09.768ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
252025-07-17T11:30:09.768ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
262025-07-17T11:30:09.768ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
272025-07-17T11:30:09.768ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 5.33585ms iteration = 27 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
282025-07-17T11:30:09.769ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
292025-07-17T11:30:09.769ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
302025-07-17T11:30:09.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
312025-07-17T11:30:09.770ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
322025-07-17T11:30:09.770ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 7.090213ms iteration = 53 status = {"chicken_switches_updated":false}
332025-07-17T11:30:09.770ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
342025-07-17T11:30:09.771ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
352025-07-17T11:30:09.772ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
362025-07-17T11:30:09.774ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
372025-07-17T11:30:09.774ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
382025-07-17T11:30:09.774ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
392025-07-17T11:30:09.785ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
402025-07-17T11:30:09.785ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.633888ms iteration = 27 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
412025-07-17T11:30:11.692ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 578 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = 0871498c-2b7b-43df-8a79-05d79c700bb9 response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
422025-07-17T11:30:14.763ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 54 reason = Timeout
432025-07-17T11:30:14.763ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
442025-07-17T11:30:14.763ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
452025-07-17T11:30:14.765ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
462025-07-17T11:30:14.767ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
472025-07-17T11:30:14.767ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
482025-07-17T11:30:14.768ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
492025-07-17T11:30:14.769ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
502025-07-17T11:30:14.769ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 6.562557ms iteration = 54 status = {"chicken_switches_updated":false}
512025-07-17T11:30:19.674ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 10 reason = Timeout
522025-07-17T11:30:19.674ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
532025-07-17T11:30:19.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
542025-07-17T11:30:19.677ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3613, tv_nsec: 430579981 } })
552025-07-17T11:30:19.677ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1186
562025-07-17T11:30:19.731ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1194 zone_address = fd00:1122:3344:101::2
572025-07-17T11:30:19.731ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
582025-07-17T11:30:19.731ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 10 reason = Timeout
592025-07-17T11:30:19.731ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
602025-07-17T11:30:19.731ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 10 reason = Timeout
612025-07-17T11:30:19.731ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "df7a4a21-a2a3-45eb-b86f-1b7031fa43c3", "content-length": "26", "date": "Thu, 17 Jul 2025 11:30:19 GMT"} })
622025-07-17T11:30:19.731ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1197 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
632025-07-17T11:30:19.732ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
642025-07-17T11:30:19.734ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
652025-07-17T11:30:19.734ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3613, tv_nsec: 488399783 } })
662025-07-17T11:30:19.735ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1186
672025-07-17T11:30:19.736ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.888994ms iteration = 10 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
682025-07-17T11:30:19.763ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 10 reason = Timeout
692025-07-17T11:30:19.763ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 10 reason = Timeout
702025-07-17T11:30:19.763ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 10 reason = Timeout
712025-07-17T11:30:19.763ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
722025-07-17T11:30:19.763ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
732025-07-17T11:30:19.763ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
742025-07-17T11:30:19.763ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 28 reason = Timeout
752025-07-17T11:30:19.763ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 10 reason = Timeout
762025-07-17T11:30:19.763ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
772025-07-17T11:30:19.763ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
782025-07-17T11:30:19.763ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 28 reason = Timeout
792025-07-17T11:30:19.763ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
802025-07-17T11:30:19.763ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 55 reason = Timeout
812025-07-17T11:30:19.763ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
822025-07-17T11:30:19.763ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 10 reason = Timeout
832025-07-17T11:30:19.763ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
842025-07-17T11:30:19.763ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 10 reason = Timeout
852025-07-17T11:30:19.763ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 10 reason = Timeout
862025-07-17T11:30:19.763ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
872025-07-17T11:30:19.763ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
882025-07-17T11:30:19.763ZDEBGnexus (BackgroundTasks): activating background_task = sp_ereport_ingester iteration = 10 reason = Timeout
892025-07-17T11:30:19.763ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 10 reason = Timeout
902025-07-17T11:30:19.763ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 10 reason = Timeout
912025-07-17T11:30:19.763ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
922025-07-17T11:30:19.763ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
932025-07-17T11:30:19.763ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 10 reason = Timeout
942025-07-17T11:30:19.763ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
952025-07-17T11:30:19.763ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
962025-07-17T11:30:19.763ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 10 reason = Timeout
972025-07-17T11:30:19.763ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
982025-07-17T11:30:19.763ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
992025-07-17T11:30:19.763ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 10 reason = Timeout
1002025-07-17T11:30:19.763ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
1012025-07-17T11:30:19.763ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 10 reason = Timeout
1022025-07-17T11:30:19.763ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
1032025-07-17T11:30:19.763ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 11 reason = Timeout
1042025-07-17T11:30:19.763ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
1052025-07-17T11:30:19.763ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
1062025-07-17T11:30:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
1072025-07-17T11:30:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
1082025-07-17T11:30:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
1092025-07-17T11:30:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
1102025-07-17T11:30:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
1112025-07-17T11:30:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
1122025-07-17T11:30:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1132025-07-17T11:30:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
1142025-07-17T11:30:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
1152025-07-17T11:30:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
1162025-07-17T11:30:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
1172025-07-17T11:30:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
1182025-07-17T11:30:19.764ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12225, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3613, tv_nsec: 517612928 } })
1192025-07-17T11:30:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
1202025-07-17T11:30:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
1212025-07-17T11:30:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
1222025-07-17T11:30:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
1232025-07-17T11:30:19.766ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 3.328444ms iteration = 10 status = {"errors":[],"garbage_collect_requested":[]}
1242025-07-17T11:30:19.767ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1252025-07-17T11:30:19.767ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1262025-07-17T11:30:19.768ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1272025-07-17T11:30:19.768ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1282025-07-17T11:30:19.769ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
1292025-07-17T11:30:19.769ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1302025-07-17T11:30:19.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
1312025-07-17T11:30:19.770ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
1322025-07-17T11:30:19.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
1332025-07-17T11:30:19.772ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
1342025-07-17T11:30:19.772ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
1352025-07-17T11:30:19.772ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 8.597983ms iteration = 10 status = {"errors":[],"finish_invoked_ok":[]}
1362025-07-17T11:30:19.772ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
1372025-07-17T11:30:19.772ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1382025-07-17T11:30:19.773ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
1392025-07-17T11:30:19.773ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
1402025-07-17T11:30:19.773ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
1412025-07-17T11:30:19.773ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
1422025-07-17T11:30:19.773ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
1432025-07-17T11:30:19.773ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1442025-07-17T11:30:19.773ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
1452025-07-17T11:30:19.774ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1462025-07-17T11:30:19.775ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 11.912557ms iteration = 28 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
1472025-07-17T11:30:19.775ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
1482025-07-17T11:30:19.775ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1492025-07-17T11:30:19.775ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
1502025-07-17T11:30:19.775ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
1512025-07-17T11:30:19.775ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1522025-07-17T11:30:19.775ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
1532025-07-17T11:30:19.775ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
1542025-07-17T11:30:19.775ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
1552025-07-17T11:30:19.775ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
1562025-07-17T11:30:19.776ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
1572025-07-17T11:30:19.777ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
1582025-07-17T11:30:19.777ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
1592025-07-17T11:30:19.777ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
1602025-07-17T11:30:19.777ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
1612025-07-17T11:30:19.777ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
1622025-07-17T11:30:19.777ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1632025-07-17T11:30:19.777ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3613, tv_nsec: 531049145 } })
1642025-07-17T11:30:19.777ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1652025-07-17T11:30:19.778ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1186 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
1662025-07-17T11:30:19.778ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
1672025-07-17T11:30:19.778ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
1682025-07-17T11:30:19.778ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
1692025-07-17T11:30:19.778ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
1702025-07-17T11:30:19.778ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
1712025-07-17T11:30:19.778ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
1722025-07-17T11:30:19.778ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 15.385482ms iteration = 10 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
1732025-07-17T11:30:19.779ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
1742025-07-17T11:30:19.779ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 16.120053ms iteration = 55 status = {"chicken_switches_updated":false}
1752025-07-17T11:30:19.779ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 16.004391ms iteration = 10 status = {"errors":[],"requests_created_ok":[]}
1762025-07-17T11:30:19.779ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1772025-07-17T11:30:19.780ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
1782025-07-17T11:30:19.780ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
1792025-07-17T11:30:19.781ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
1802025-07-17T11:30:19.781ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1812025-07-17T11:30:19.781ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
1822025-07-17T11:30:19.784ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
1832025-07-17T11:30:19.784ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
1842025-07-17T11:30:19.784ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
1852025-07-17T11:30:19.784ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
1862025-07-17T11:30:19.784ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
1872025-07-17T11:30:19.784ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
1882025-07-17T11:30:19.787ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
1892025-07-17T11:30:19.787ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
1902025-07-17T11:30:19.794ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1912025-07-17T11:30:19.797ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1922025-07-17T11:30:19.798ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
1932025-07-17T11:30:19.798ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
1942025-07-17T11:30:19.799ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
1952025-07-17T11:30:19.800ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
1962025-07-17T11:30:19.800ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
1972025-07-17T11:30:19.800ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
1982025-07-17T11:30:19.810ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
1992025-07-17T11:30:19.810ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 46.676115ms iteration = 28 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
2002025-07-17T11:30:19.817ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
2012025-07-17T11:30:19.817ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
2022025-07-17T11:30:19.820ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 145.67382ms iteration = 10 status = null
2032025-07-17T11:30:19.824ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1194 zone_address = fd00:1122:3344:101::2
2042025-07-17T11:30:19.824ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
2052025-07-17T11:30:19.825ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "36a466da-25f9-49a0-b738-dee43684609b", "content-length": "26", "date": "Thu, 17 Jul 2025 11:30:19 GMT"} })
2062025-07-17T11:30:19.825ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1197 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
2072025-07-17T11:30:19.826ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
2082025-07-17T11:30:19.844ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
2092025-07-17T11:30:19.844ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
2102025-07-17T11:30:19.846ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
2112025-07-17T11:30:19.846ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
2122025-07-17T11:30:19.846ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
2132025-07-17T11:30:19.846ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
2142025-07-17T11:30:19.846ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 83.385219ms iteration = 10 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
2152025-07-17T11:30:19.848ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
2162025-07-17T11:30:19.848ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
2172025-07-17T11:30:19.848ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
2182025-07-17T11:30:19.848ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
2192025-07-17T11:30:19.849ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 86.372039ms iteration = 10 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
2202025-07-17T11:30:19.859ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
2212025-07-17T11:30:19.860ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
2222025-07-17T11:30:19.863ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
2232025-07-17T11:30:19.863ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
2242025-07-17T11:30:19.864ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2252025-07-17T11:30:19.865ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:550 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
2262025-07-17T11:30:19.865ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 102.39204ms iteration = 10 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
2272025-07-17T11:30:19.869ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 0ae78f24-48db-4cf7-91cb-8d5311f2a17f, lookup_type: ById(0ae78f24-48db-4cf7-91cb-8d5311f2a17f) } result = Ok(())
2282025-07-17T11:30:19.869ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
2292025-07-17T11:30:19.869ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
2302025-07-17T11:30:19.870ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:372
2312025-07-17T11:30:19.870ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 107.069801ms iteration = 10 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
2322025-07-17T11:30:19.871ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 107.833112ms iteration = 10 status = 0
2332025-07-17T11:30:19.877ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 113.665759ms iteration = 11 status = {"physical_disks_added":0}
2342025-07-17T11:30:19.889ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/all-sp-ids
2352025-07-17T11:30:19.889ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/all-sp-ids", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5b56237-e875-4d94-9baa-bb1e0f9c8290", "content-length": "78", "date": "Thu, 17 Jul 2025 11:30:19 GMT"} })
2362025-07-17T11:30:19.889ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = switch
2372025-07-17T11:30:19.889ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = sled
2382025-07-17T11:30:19.889ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 1 sp_type = sled
2392025-07-17T11:30:19.890ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 1 sp_type = sled
2402025-07-17T11:30:19.890ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = switch
2412025-07-17T11:30:19.890ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = sled
2422025-07-17T11:30:19.891ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 1 sp_type = sled
2432025-07-17T11:30:19.892ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 0 sp_type = switch
2442025-07-17T11:30:19.892ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 0 sp_type = sled
2452025-07-17T11:30:19.896ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 1 sp_type = sled
2462025-07-17T11:30:19.896ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 1 sp_type = sled
2472025-07-17T11:30:19.896ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 1 sp_type = sled
2482025-07-17T11:30:19.896ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 0 sp_type = sled
2492025-07-17T11:30:19.896ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = sled
2502025-07-17T11:30:19.896ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 0 sp_type = switch
2512025-07-17T11:30:19.896ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = switch
2522025-07-17T11:30:19.896ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = sled
2532025-07-17T11:30:19.896ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = switch
2542025-07-17T11:30:19.898ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 1 sp_type = sled start_ena = None
2552025-07-17T11:30:19.898ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/1/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
2562025-07-17T11:30:19.898ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/1/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f97e53b-77cb-4a37-9b97-41005bbf60b5", "content-length": "81", "date": "Thu, 17 Jul 2025 11:30:19 GMT"} })
2572025-07-17T11:30:19.904ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
2582025-07-17T11:30:19.905ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
2592025-07-17T11:30:19.905ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 141.489295ms iteration = 10 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
2602025-07-17T11:30:19.941ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 0 sp_type = sled start_ena = None
2612025-07-17T11:30:19.941ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
2622025-07-17T11:30:19.942ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "958fa14f-d052-47d2-9def-00dbd8c1c234", "content-length": "81", "date": "Thu, 17 Jul 2025 11:30:19 GMT"} })
2632025-07-17T11:30:19.946ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 0 sp_type = switch start_ena = None
2642025-07-17T11:30:19.946ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
2652025-07-17T11:30:19.946ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc1bfb5f-0614-4d5d-823c-33d9d602ba50", "content-length": "81", "date": "Thu, 17 Jul 2025 11:30:19 GMT"} })
2662025-07-17T11:30:19.947ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1194 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 zone_address = fd00:1122:3344:101::2
2672025-07-17T11:30:19.947ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
2682025-07-17T11:30:19.947ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "a5aaf848-0f88-49ed-a7f8-a5502cddf21e", "content-length": "26", "date": "Thu, 17 Jul 2025 11:30:19 GMT"} })
2692025-07-17T11:30:19.947ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1197 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
2702025-07-17T11:30:19.949ZDEBGnexus (BackgroundTasks): activation complete background_task = sp_ereport_ingester elapsed = 185.955572ms iteration = 10 status = {"errors":[],"sps":[]}
2712025-07-17T11:30:19.950ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
2722025-07-17T11:30:19.975ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 244.093648ms iteration = 10 status = {}
2732025-07-17T11:30:19.976ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
2742025-07-17T11:30:19.976ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
2752025-07-17T11:30:19.978ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2762025-07-17T11:30:19.980ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
2772025-07-17T11:30:19.980ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
2782025-07-17T11:30:19.980ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
2792025-07-17T11:30:19.982ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2802025-07-17T11:30:19.984ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
2812025-07-17T11:30:19.984ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
2822025-07-17T11:30:19.984ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
2832025-07-17T11:30:20.005ZINFOnexus (BackgroundTasks): VPC route manager sled 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
2842025-07-17T11:30:20.005ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
2852025-07-17T11:30:20.006ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "81b82a88-5d56-4c7e-832b-0a2b19a42c14", "content-length": "372", "date": "Thu, 17 Jul 2025 11:30:19 GMT"} })
2862025-07-17T11:30:20.006ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
2872025-07-17T11:30:20.006ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
2882025-07-17T11:30:20.061ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
2892025-07-17T11:30:20.061ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "58a59fb0-af6d-415c-a885-7b7c6f5625a0", "content-length": "691", "date": "Thu, 17 Jul 2025 11:30:19 GMT"} })
2902025-07-17T11:30:20.062ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
2912025-07-17T11:30:20.062ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
2922025-07-17T11:30:20.062ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
2932025-07-17T11:30:20.063ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
2942025-07-17T11:30:20.063ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
2952025-07-17T11:30:20.063ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 300.381289ms iteration = 10 status = {}
2962025-07-17T11:30:20.063ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "119d1b15-4d83-4d8b-b4a5-93128560d756", "date": "Thu, 17 Jul 2025 11:30:19 GMT"} })
2972025-07-17T11:30:20.063ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
2982025-07-17T11:30:20.064ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
2992025-07-17T11:30:20.066ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
3002025-07-17T11:30:20.066ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
3012025-07-17T11:30:20.069ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
3022025-07-17T11:30:20.069ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
3032025-07-17T11:30:20.073ZINFOnexus (BackgroundTasks): VPC route manager sled 0ae78f24-48db-4cf7-91cb-8d5311f2a17f push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
3042025-07-17T11:30:20.073ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
3052025-07-17T11:30:20.073ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "f041d2b0-dedd-4315-ad04-ce7fa6a902da", "date": "Thu, 17 Jul 2025 11:30:19 GMT"} })
3062025-07-17T11:30:20.074ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 311.254873ms iteration = 10 status = {}
3072025-07-17T11:30:20.103ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
3082025-07-17T11:30:20.103ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
3092025-07-17T11:30:20.104ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 151caa7c-6a05-4995-94b4-7946d428117d, rack_id: 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5, switch_location: "switch0", port_name: Name(Name("qsfp0")), port_settings_id: Some(2e30c7ae-26da-46e4-9127-aac194921f20) } rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0
3102025-07-17T11:30:20.104ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
3112025-07-17T11:30:20.105ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
3122025-07-17T11:30:20.118ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1832 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = Name(Name("qsfp0"))
3132025-07-17T11:30:20.118ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1859 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0 switch_port_id = Name(Name("qsfp0")) switch_port_settings_id = 2e30c7ae-26da-46e4-9127-aac194921f20
3142025-07-17T11:30:20.118ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
3152025-07-17T11:30:20.118ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
3162025-07-17T11:30:20.118ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
3172025-07-17T11:30:20.118ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
3182025-07-17T11:30:20.118ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
3192025-07-17T11:30:20.118ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2006 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
3202025-07-17T11:30:20.118ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
3212025-07-17T11:30:20.118ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2042 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
3222025-07-17T11:30:20.118ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
3232025-07-17T11:30:20.118ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
3242025-07-17T11:30:20.119ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
3252025-07-17T11:30:20.121ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0
3262025-07-17T11:30:20.121ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
3272025-07-17T11:30:20.432ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "1d387670-42ae-4a0e-bde3-57c94630e975", "date": "Thu, 17 Jul 2025 11:30:20 GMT"} })
3282025-07-17T11:30:20.432ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch = Switch0
3292025-07-17T11:30:20.432ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
3302025-07-17T11:30:20.432ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b80ea33-33a0-47e6-986b-d406a91ca8a7", "content-length": "682", "date": "Thu, 17 Jul 2025 11:30:19 GMT"} })
3312025-07-17T11:30:20.432ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
3322025-07-17T11:30:20.432ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
3332025-07-17T11:30:20.436ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
3342025-07-17T11:30:20.436ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
3352025-07-17T11:30:20.440ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
3362025-07-17T11:30:20.441ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
3372025-07-17T11:30:20.442ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
3382025-07-17T11:30:20.443ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
3392025-07-17T11:30:20.449ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 686.439759ms iteration = 10 status = {}
3402025-07-17T11:30:21.690ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 345 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = 6e6284e1-bb21-4e2c-bcca-31c307ce4a51 response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
3412025-07-17T11:30:24.762ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 56 reason = Timeout
3422025-07-17T11:30:24.763ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
3432025-07-17T11:30:24.763ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
3442025-07-17T11:30:24.765ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3452025-07-17T11:30:24.768ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
3462025-07-17T11:30:24.768ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
3472025-07-17T11:30:24.768ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
3482025-07-17T11:30:24.770ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
3492025-07-17T11:30:24.770ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 7.53204ms iteration = 56 status = {"chicken_switches_updated":false}
3502025-07-17T11:30:29.762ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 29 reason = Timeout
3512025-07-17T11:30:29.762ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 29 reason = Timeout
3522025-07-17T11:30:29.762ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
3532025-07-17T11:30:29.762ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 57 reason = Timeout
3542025-07-17T11:30:29.762ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3552025-07-17T11:30:29.762ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
3562025-07-17T11:30:29.763ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3572025-07-17T11:30:29.763ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
3582025-07-17T11:30:29.763ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
3592025-07-17T11:30:29.764ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3602025-07-17T11:30:29.765ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3612025-07-17T11:30:29.765ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
3622025-07-17T11:30:29.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
3632025-07-17T11:30:29.768ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
3642025-07-17T11:30:29.768ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3652025-07-17T11:30:29.768ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3662025-07-17T11:30:29.768ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
3672025-07-17T11:30:29.768ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
3682025-07-17T11:30:29.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
3692025-07-17T11:30:29.770ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3702025-07-17T11:30:29.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3712025-07-17T11:30:29.770ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
3722025-07-17T11:30:29.770ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 8.078966ms iteration = 57 status = {"chicken_switches_updated":false}
3732025-07-17T11:30:29.771ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3742025-07-17T11:30:29.774ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
3752025-07-17T11:30:29.774ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3762025-07-17T11:30:29.774ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3772025-07-17T11:30:29.783ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
3782025-07-17T11:30:29.783ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.869665ms iteration = 29 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
3792025-07-17T11:30:29.799ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 37.11785ms iteration = 29 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
3802025-07-17T11:30:31.691ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 375 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = ff833d85-bae3-495a-b702-07a7360e6997 response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
3812025-07-17T11:30:34.763ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 58 reason = Timeout
3822025-07-17T11:30:34.763ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
3832025-07-17T11:30:34.763ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
3842025-07-17T11:30:34.765ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3852025-07-17T11:30:34.768ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
3862025-07-17T11:30:34.768ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
3872025-07-17T11:30:34.768ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
3882025-07-17T11:30:34.769ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
3892025-07-17T11:30:34.769ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 6.538926ms iteration = 58 status = {"chicken_switches_updated":false}
3902025-07-17T11:30:39.763ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 30 reason = Timeout
3912025-07-17T11:30:39.763ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 30 reason = Timeout
3922025-07-17T11:30:39.763ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
3932025-07-17T11:30:39.763ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3942025-07-17T11:30:39.763ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 59 reason = Timeout
3952025-07-17T11:30:39.763ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
3962025-07-17T11:30:39.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3972025-07-17T11:30:39.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
3982025-07-17T11:30:39.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
3992025-07-17T11:30:39.766ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4002025-07-17T11:30:39.766ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
4012025-07-17T11:30:39.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
4022025-07-17T11:30:39.766ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4032025-07-17T11:30:39.768ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.184915ms iteration = 30 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
4042025-07-17T11:30:39.769ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
4052025-07-17T11:30:39.769ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4062025-07-17T11:30:39.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4072025-07-17T11:30:39.770ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
4082025-07-17T11:30:39.770ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
4092025-07-17T11:30:39.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
4102025-07-17T11:30:39.771ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4112025-07-17T11:30:39.771ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4122025-07-17T11:30:39.771ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
4132025-07-17T11:30:39.771ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 7.848553ms iteration = 59 status = {"chicken_switches_updated":false}
4142025-07-17T11:30:39.772ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4152025-07-17T11:30:39.775ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
4162025-07-17T11:30:39.775ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4172025-07-17T11:30:39.775ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4182025-07-17T11:30:39.784ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
4192025-07-17T11:30:39.784ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.739173ms iteration = 30 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
4202025-07-17T11:30:41.691ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 328 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = 04995d05-c9eb-4f95-b6db-6828db07ab99 response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
4212025-07-17T11:30:44.763ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 60 reason = Timeout
4222025-07-17T11:30:44.763ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
4232025-07-17T11:30:44.763ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
4242025-07-17T11:30:44.765ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4252025-07-17T11:30:44.768ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
4262025-07-17T11:30:44.768ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
4272025-07-17T11:30:44.768ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
4282025-07-17T11:30:44.770ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
4292025-07-17T11:30:44.770ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 6.865291ms iteration = 60 status = {"chicken_switches_updated":false}
4302025-07-17T11:30:46.683ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::d]:62728
4312025-07-17T11:30:46.684ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62728 req_id = 60b7baa6-4e7a-40fc-b1ea-56eb17c3e7fe roles = RoleSet { roles: {} } uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?sort_by=id_ascending
4322025-07-17T11:30:46.684ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62728 req_id = 60b7baa6-4e7a-40fc-b1ea-56eb17c3e7fe resource = Database result = Ok(()) uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?sort_by=id_ascending
4332025-07-17T11:30:46.686ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 2742 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62728 req_id = 60b7baa6-4e7a-40fc-b1ea-56eb17c3e7fe response_code = 200 uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?sort_by=id_ascending
4342025-07-17T11:30:46.687ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62728 req_id = 29df20d0-562e-4700-ac00-df42107e8ab3 roles = RoleSet { roles: {} } uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmMzlkM2U3MC02NmU0LTQ2OGEtYTQ0ZC03NDZkMjJiZWEzYTAifX0%3D
4352025-07-17T11:30:46.687ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62728 req_id = 29df20d0-562e-4700-ac00-df42107e8ab3 resource = Database result = Ok(()) uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmMzlkM2U3MC02NmU0LTQ2OGEtYTQ0ZC03NDZkMjJiZWEzYTAifX0%3D
4362025-07-17T11:30:46.690ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 3847 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62728 req_id = 29df20d0-562e-4700-ac00-df42107e8ab3 response_code = 200 uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmMzlkM2U3MC02NmU0LTQ2OGEtYTQ0ZC03NDZkMjJiZWEzYTAifX0%3D
4372025-07-17T11:30:49.675ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 6 reason = Timeout
4382025-07-17T11:30:49.675ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 6 reason = Timeout
4392025-07-17T11:30:49.675ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
4402025-07-17T11:30:49.675ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 8 reason = Timeout
4412025-07-17T11:30:49.675ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
4422025-07-17T11:30:49.675ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 8 reason = Timeout
4432025-07-17T11:30:49.675ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 11 reason = Timeout
4442025-07-17T11:30:49.675ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
4452025-07-17T11:30:49.675ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 6 reason = Timeout
4462025-07-17T11:30:49.675ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-07-17 11:20:49.675699964 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
4472025-07-17T11:30:49.675ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 6 reason = Timeout
4482025-07-17T11:30:49.675ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
4492025-07-17T11:30:49.675ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
4502025-07-17T11:30:49.675ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
4512025-07-17T11:30:49.675ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 6 reason = Timeout
4522025-07-17T11:30:49.675ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 6 reason = Timeout
4532025-07-17T11:30:49.675ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
4542025-07-17T11:30:49.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
4552025-07-17T11:30:49.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
4562025-07-17T11:30:49.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
4572025-07-17T11:30:49.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
4582025-07-17T11:30:49.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
4592025-07-17T11:30:49.676ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("3144a75a-317f-4b68-93e2-a210ca1189d1.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("87408888-9df4-4e0c-bc8e-2f920d039100.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("ee4378c5-e5f2-41cf-8969-e5e03e1b9a41.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("ee4378c5-e5f2-41cf-8969-e5e03e1b9a41.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:3::1)) }], valid_until: Instant { tv_sec: 3643, tv_nsec: 429814478 } })
4602025-07-17T11:30:49.676ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("3dd3bf3b-7760-4750-b1d3-52ee777335f0.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("79e5c77d-18b3-4951-8d11-cfa3eae2b3e3.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("79e5c77d-18b3-4951-8d11-cfa3eae2b3e3.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::8)) }], valid_until: Instant { tv_sec: 3643, tv_nsec: 429805048 } })
4612025-07-17T11:30:49.676ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
4622025-07-17T11:30:49.677ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
4632025-07-17T11:30:49.677ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
4642025-07-17T11:30:49.677ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.46566ms iteration = 6 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
4652025-07-17T11:30:49.677ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } dns_group = internal
4662025-07-17T11:30:49.677ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.365168ms iteration = 6 status = {"addresses":["[fd00:1122:3344:1::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:3::1]:5353"]}
4672025-07-17T11:30:49.678ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 7 reason = Timeout
4682025-07-17T11:30:49.678ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:85 reason = blueprint disabled target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a
4692025-07-17T11:30:49.678ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 17.17\u{b5}s iteration = 7 status = {"enabled":false,"target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a"}
4702025-07-17T11:30:49.678ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
4712025-07-17T11:30:49.678ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
4722025-07-17T11:30:49.679ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3643, tv_nsec: 432671416 } })
4732025-07-17T11:30:49.679ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1186
4742025-07-17T11:30:49.680ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-07-17 11:20:49.675699964 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
4752025-07-17T11:30:49.680ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 4.384598ms iteration = 6 status = {"expiration":"2025-07-17T11:20:49.675699964Z","pruned":[]}
4762025-07-17T11:30:49.680ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4772025-07-17T11:30:49.681ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4782025-07-17T11:30:49.682ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4792025-07-17T11:30:49.684ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
4802025-07-17T11:30:49.684ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
4812025-07-17T11:30:49.684ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
4822025-07-17T11:30:49.684ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 3 reason = Timeout
4832025-07-17T11:30:49.684ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_planner iteration = 8 reason = Timeout
4842025-07-17T11:30:49.684ZDEBGnexus (BackgroundTasks): activating background_task = service_firewall_rule_propagation iteration = 2 reason = Timeout
4852025-07-17T11:30:49.684ZDEBGnexus (service-firewall-rule-progator): starting background task for service firewall rule propagation background_task = service_firewall_rule_propagation
4862025-07-17T11:30:49.684ZDEBGnexus (BackgroundTasks): blueprint planning disabled, doing nothing background_task = blueprint_planner
4872025-07-17T11:30:49.684ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_planner elapsed = 7.97\u{b5}s iteration = 8 status = "Disabled"
4882025-07-17T11:30:49.684ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
4892025-07-17T11:30:49.684ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
4902025-07-17T11:30:49.684ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 6 reason = Timeout
4912025-07-17T11:30:49.684ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 6 reason = Timeout
4922025-07-17T11:30:49.684ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:53
4932025-07-17T11:30:49.684ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
4942025-07-17T11:30:49.684ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
4952025-07-17T11:30:49.684ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
4962025-07-17T11:30:49.685ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
4972025-07-17T11:30:49.685ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
4982025-07-17T11:30:49.685ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
4992025-07-17T11:30:49.686ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 6 reason = Timeout
5002025-07-17T11:30:49.686ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
5012025-07-17T11:30:49.686ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
5022025-07-17T11:30:49.687ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
5032025-07-17T11:30:49.687ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
5042025-07-17T11:30:49.687ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
5052025-07-17T11:30:49.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
5062025-07-17T11:30:49.689ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
5072025-07-17T11:30:49.689ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
5082025-07-17T11:30:49.689ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
5092025-07-17T11:30:49.689ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
5102025-07-17T11:30:49.690ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
5112025-07-17T11:30:49.691ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5122025-07-17T11:30:49.691ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5132025-07-17T11:30:49.691ZDEBGnexus (BackgroundTasks): activating background_task = tuf_artifact_replication iteration = 2 reason = Timeout
5142025-07-17T11:30:49.691ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
5152025-07-17T11:30:49.691ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 6 reason = Timeout
5162025-07-17T11:30:49.691ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
5172025-07-17T11:30:49.692ZDEBGnexus (BackgroundTasks): activating background_task = webhook_deliverator iteration = 6 reason = Timeout
5182025-07-17T11:30:49.692ZDEBGnexus (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} }
5192025-07-17T11:30:49.692ZDEBGnexus (BackgroundTasks): activating background_task = alert_dispatcher iteration = 6 reason = Timeout
5202025-07-17T11:30:49.692ZDEBGnexus (BackgroundTasks): roles background_task = alert_dispatcher roles = RoleSet { roles: {} }
5212025-07-17T11:30:49.692ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
5222025-07-17T11:30:49.692ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = alert_dispatcher resource = Database result = Ok(())
5232025-07-17T11:30:49.692ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database result = Ok(())
5242025-07-17T11:30:49.692ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
5252025-07-17T11:30:49.693ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
5262025-07-17T11:30:49.693ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
5272025-07-17T11:30:49.693ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5282025-07-17T11:30:49.694ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5292025-07-17T11:30:49.694ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
5302025-07-17T11:30:49.694ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:170 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
5312025-07-17T11:30:49.694ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 9.196901ms iteration = 6 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
5322025-07-17T11:30:49.695ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5332025-07-17T11:30:49.696ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
5342025-07-17T11:30:49.696ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
5352025-07-17T11:30:49.697ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
5362025-07-17T11:30:49.697ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 5.188708ms iteration = 6 status = {"errors":[],"found_port_ok":[]}
5372025-07-17T11:30:49.697ZINFOnexus (BackgroundTasks): all webhook delivery tasks completed background_task = webhook_deliverator file = nexus/src/app/background/tasks/webhook_deliverator.rs:204 num_receivers = 0
5382025-07-17T11:30:49.697ZDEBGnexus (BackgroundTasks): activation complete background_task = webhook_deliverator elapsed = 5.188578ms iteration = 6 status = {"by_rx":{},"error":null}
5392025-07-17T11:30:49.697ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Inventory result = Ok(())
5402025-07-17T11:30:49.697ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
5412025-07-17T11:30:49.697ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
5422025-07-17T11:30:49.699ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
5432025-07-17T11:30:49.699ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
5442025-07-17T11:30:49.699ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
5452025-07-17T11:30:49.700ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
5462025-07-17T11:30:49.700ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
5472025-07-17T11:30:49.700ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
5482025-07-17T11:30:49.700ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
5492025-07-17T11:30:49.700ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
5502025-07-17T11:30:49.700ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
5512025-07-17T11:30:49.700ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
5522025-07-17T11:30:49.700ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
5532025-07-17T11:30:49.700ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
5542025-07-17T11:30:49.700ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
5552025-07-17T11:30:49.701ZDEBGnexus (BackgroundTasks): roles background_task = alert_dispatcher roles = RoleSet { roles: {} }
5562025-07-17T11:30:49.701ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = alert_dispatcher resource = Database result = Ok(())
5572025-07-17T11:30:49.702ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5582025-07-17T11:30:49.704ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
5592025-07-17T11:30:49.705ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
5602025-07-17T11:30:49.705ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
5612025-07-17T11:30:49.705ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
5622025-07-17T11:30:49.706ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
5632025-07-17T11:30:49.707ZDEBGnexus (BackgroundTasks): activation complete background_task = alert_dispatcher elapsed = 15.579736ms iteration = 6 status = {"dispatched":[],"errors":[],"glob_version":"161.0.0","globs_reprocessed":{},"no_receivers":[]}
5642025-07-17T11:30:49.708ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
5652025-07-17T11:30:49.708ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
5662025-07-17T11:30:49.717ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
5672025-07-17T11:30:49.718ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
5682025-07-17T11:30:49.732ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 11 reason = Timeout
5692025-07-17T11:30:49.732ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 11 reason = Timeout
5702025-07-17T11:30:49.732ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
5712025-07-17T11:30:49.733ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
5722025-07-17T11:30:49.738ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5732025-07-17T11:30:49.743ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3643, tv_nsec: 496871352 } })
5742025-07-17T11:30:49.744ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1186
5752025-07-17T11:30:49.745ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 12.382953ms iteration = 11 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
5762025-07-17T11:30:49.745ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
5772025-07-17T11:30:49.745ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
5782025-07-17T11:30:49.746ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
5792025-07-17T11:30:49.748ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
5802025-07-17T11:30:49.749ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
5812025-07-17T11:30:49.752ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
5822025-07-17T11:30:49.752ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
5832025-07-17T11:30:49.763ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 11 reason = Timeout
5842025-07-17T11:30:49.763ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
5852025-07-17T11:30:49.763ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
5862025-07-17T11:30:49.763ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 11 reason = Timeout
5872025-07-17T11:30:49.763ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
5882025-07-17T11:30:49.763ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
5892025-07-17T11:30:49.763ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 11 reason = Timeout
5902025-07-17T11:30:49.763ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
5912025-07-17T11:30:49.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
5922025-07-17T11:30:49.764ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 11 reason = Timeout
5932025-07-17T11:30:49.764ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
5942025-07-17T11:30:49.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
5952025-07-17T11:30:49.764ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 11 reason = Timeout
5962025-07-17T11:30:49.764ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
5972025-07-17T11:30:49.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
5982025-07-17T11:30:49.764ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 11 reason = Timeout
5992025-07-17T11:30:49.764ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
6002025-07-17T11:30:49.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
6012025-07-17T11:30:49.764ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 11 reason = Timeout
6022025-07-17T11:30:49.764ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
6032025-07-17T11:30:49.765ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 11 reason = Timeout
6042025-07-17T11:30:49.765ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 12 reason = Timeout
6052025-07-17T11:30:49.765ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
6062025-07-17T11:30:49.765ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
6072025-07-17T11:30:49.765ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 31 reason = Timeout
6082025-07-17T11:30:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
6092025-07-17T11:30:49.765ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
6102025-07-17T11:30:49.765ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 31 reason = Timeout
6112025-07-17T11:30:49.765ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6122025-07-17T11:30:49.765ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 11 reason = Timeout
6132025-07-17T11:30:49.765ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
6142025-07-17T11:30:49.765ZDEBGnexus (BackgroundTasks): activating background_task = sp_ereport_ingester iteration = 11 reason = Timeout
6152025-07-17T11:30:49.765ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
6162025-07-17T11:30:49.765ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 11 reason = Timeout
6172025-07-17T11:30:49.765ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
6182025-07-17T11:30:49.765ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
6192025-07-17T11:30:49.765ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 11 reason = Timeout
6202025-07-17T11:30:49.765ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
6212025-07-17T11:30:49.765ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 11 reason = Timeout
6222025-07-17T11:30:49.765ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
6232025-07-17T11:30:49.765ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 61 reason = Timeout
6242025-07-17T11:30:49.765ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
6252025-07-17T11:30:49.765ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 11 reason = Timeout
6262025-07-17T11:30:49.765ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
6272025-07-17T11:30:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6282025-07-17T11:30:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
6292025-07-17T11:30:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
6302025-07-17T11:30:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
6312025-07-17T11:30:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
6322025-07-17T11:30:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
6332025-07-17T11:30:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
6342025-07-17T11:30:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
6352025-07-17T11:30:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
6362025-07-17T11:30:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
6372025-07-17T11:30:49.765ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12225, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3643, tv_nsec: 519245157 } })
6382025-07-17T11:30:49.783ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6392025-07-17T11:30:49.788ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
6402025-07-17T11:30:49.788ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6412025-07-17T11:30:49.788ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6422025-07-17T11:30:49.796ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6432025-07-17T11:30:49.796ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6442025-07-17T11:30:49.797ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
6452025-07-17T11:30:49.797ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:138 found = 0 not_ready_to_be_deleted = 0
6462025-07-17T11:30:49.797ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 112.763157ms iteration = 6 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
6472025-07-17T11:30:49.799ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6482025-07-17T11:30:49.799ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6492025-07-17T11:30:49.801ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
6502025-07-17T11:30:49.801ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
6512025-07-17T11:30:49.801ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
6522025-07-17T11:30:49.801ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6532025-07-17T11:30:49.802ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6542025-07-17T11:30:49.802ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6552025-07-17T11:30:49.803ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
6562025-07-17T11:30:49.803ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
6572025-07-17T11:30:49.803ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
6582025-07-17T11:30:49.804ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
6592025-07-17T11:30:49.804ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
6602025-07-17T11:30:49.804ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
6612025-07-17T11:30:49.805ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
6622025-07-17T11:30:49.805ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
6632025-07-17T11:30:49.805ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
6642025-07-17T11:30:49.805ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
6652025-07-17T11:30:49.805ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
6662025-07-17T11:30:49.806ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
6672025-07-17T11:30:49.806ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
6682025-07-17T11:30:49.806ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
6692025-07-17T11:30:49.808ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6702025-07-17T11:30:49.810ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 46.075717ms iteration = 11 status = {"errors":[],"requests_created_ok":[]}
6712025-07-17T11:30:49.810ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
6722025-07-17T11:30:49.810ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
6732025-07-17T11:30:49.810ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
6742025-07-17T11:30:49.810ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
6752025-07-17T11:30:49.813ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
6762025-07-17T11:30:49.813ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
6772025-07-17T11:30:49.813ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
6782025-07-17T11:30:49.813ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6792025-07-17T11:30:49.814ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6802025-07-17T11:30:49.819ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
6812025-07-17T11:30:49.819ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
6822025-07-17T11:30:49.823ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
6832025-07-17T11:30:49.823ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 915fbba0-947f-4bf0-9f9f-48a79df81f7b dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
6842025-07-17T11:30:49.823ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
6852025-07-17T11:30:49.823ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
6862025-07-17T11:30:49.824ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3643, tv_nsec: 577932271 } })
6872025-07-17T11:30:49.824ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6882025-07-17T11:30:49.825ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1186 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
6892025-07-17T11:30:49.825ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6902025-07-17T11:30:49.826ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
6912025-07-17T11:30:49.826ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 60.871222ms iteration = 61 status = {"chicken_switches_updated":false}
6922025-07-17T11:30:49.828ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1194 zone_address = fd00:1122:3344:101::2
6932025-07-17T11:30:49.828ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
6942025-07-17T11:30:49.828ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
6952025-07-17T11:30:49.828ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
6962025-07-17T11:30:49.828ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "a163c6ed-5f2c-40c1-b519-1b8f202b17f3", "content-length": "26", "date": "Thu, 17 Jul 2025 11:30:49 GMT"} })
6972025-07-17T11:30:49.828ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1197 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
6982025-07-17T11:30:49.828ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
6992025-07-17T11:30:49.830ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
7002025-07-17T11:30:49.830ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
7012025-07-17T11:30:49.830ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
7022025-07-17T11:30:49.830ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
7032025-07-17T11:30:49.847ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7042025-07-17T11:30:49.851ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7052025-07-17T11:30:49.851ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
7062025-07-17T11:30:49.851ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
7072025-07-17T11:30:49.852ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
7082025-07-17T11:30:49.856ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
7092025-07-17T11:30:49.856ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
7102025-07-17T11:30:49.857ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
7112025-07-17T11:30:49.885ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
7122025-07-17T11:30:49.885ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
7132025-07-17T11:30:49.885ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
7142025-07-17T11:30:49.885ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
7152025-07-17T11:30:49.885ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 122.271652ms iteration = 11 status = {"errors":[],"garbage_collect_requested":[]}
7162025-07-17T11:30:49.886ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
7172025-07-17T11:30:49.889ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
7182025-07-17T11:30:49.889ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
7192025-07-17T11:30:49.889ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 915fbba0-947f-4bf0-9f9f-48a79df81f7b dns_zone_name = control-plane.oxide.internal found_so_far = 59 version = 1
7202025-07-17T11:30:49.889ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
7212025-07-17T11:30:49.889ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-07-17 11:26:34.922237 UTC dns_group = internal generation = 1
7222025-07-17T11:30:49.889ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 214.306755ms iteration = 6 status = {"generation":1}
7232025-07-17T11:30:49.892ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7242025-07-17T11:30:49.896ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
7252025-07-17T11:30:49.896ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
7262025-07-17T11:30:49.896ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
7272025-07-17T11:30:49.899ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
7282025-07-17T11:30:49.899ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
7292025-07-17T11:30:49.900ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7302025-07-17T11:30:49.901ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
7312025-07-17T11:30:49.902ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
7322025-07-17T11:30:49.902ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
7332025-07-17T11:30:49.902ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
7342025-07-17T11:30:49.905ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
7352025-07-17T11:30:49.905ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
7362025-07-17T11:30:49.905ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
7372025-07-17T11:30:49.906ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
7382025-07-17T11:30:49.906ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
7392025-07-17T11:30:49.908ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 144.455215ms iteration = 11 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
7402025-07-17T11:30:49.911ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
7412025-07-17T11:30:49.911ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 146.249128ms iteration = 31 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
7422025-07-17T11:30:49.913ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
7432025-07-17T11:30:49.914ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
7442025-07-17T11:30:49.916ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7452025-07-17T11:30:49.918ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7462025-07-17T11:30:49.920ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
7472025-07-17T11:30:49.920ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
7482025-07-17T11:30:49.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
7492025-07-17T11:30:49.922ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
7502025-07-17T11:30:49.922ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
7512025-07-17T11:30:49.923ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
7522025-07-17T11:30:49.937ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
7532025-07-17T11:30:49.938ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
7542025-07-17T11:30:49.938ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 174.859226ms iteration = 11 status = {"errors":[],"finish_invoked_ok":[]}
7552025-07-17T11:30:49.939ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7562025-07-17T11:30:49.943ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
7572025-07-17T11:30:49.943ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
7582025-07-17T11:30:49.943ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
7592025-07-17T11:30:49.944ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
7602025-07-17T11:30:49.945ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
7612025-07-17T11:30:49.945ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7622025-07-17T11:30:49.945ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7632025-07-17T11:30:49.947ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
7642025-07-17T11:30:49.948ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
7652025-07-17T11:30:49.951ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7662025-07-17T11:30:49.951ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7672025-07-17T11:30:49.954ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7682025-07-17T11:30:49.955ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7692025-07-17T11:30:49.955ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7702025-07-17T11:30:49.957ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7712025-07-17T11:30:49.958ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7722025-07-17T11:30:49.958ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
7732025-07-17T11:30:49.958ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
7742025-07-17T11:30:49.958ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
7752025-07-17T11:30:49.960ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7762025-07-17T11:30:49.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7772025-07-17T11:30:49.967ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 202.873924ms iteration = 12 status = {"physical_disks_added":0}
7782025-07-17T11:30:49.968ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7792025-07-17T11:30:49.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7802025-07-17T11:30:49.969ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
7812025-07-17T11:30:49.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
7822025-07-17T11:30:49.973ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
7832025-07-17T11:30:49.973ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7842025-07-17T11:30:49.973ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
7852025-07-17T11:30:49.974ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7862025-07-17T11:30:49.975ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-07-17T11:25:46.964149Z, time_modified: 2025-07-17T11:25:46.964149Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 1fa836d9-7854-4ada-8c89-c26d922646c6, db_silo: Silo { identity: SiloIdentity { id: 1fa836d9-7854-4ada-8c89-c26d922646c6, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-07-17T11:26:35.643175Z, time_modified: 2025-07-17T11:26:35.643175Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "d7c3455b70a66c86ec2a7e597ca46fe0ea0c49dde5c1363f198c20aa77d58442" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 1fa836d9-7854-4ada-8c89-c26d922646c6, db_silo: Silo { identity: SiloIdentity { id: 1fa836d9-7854-4ada-8c89-c26d922646c6, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-07-17T11:26:35.643175Z, time_modified: 2025-07-17T11:26:35.643175Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "d7c3455b70a66c86ec2a7e597ca46fe0ea0c49dde5c1363f198c20aa77d58442" }] }) }
7872025-07-17T11:30:49.975ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 299.840153ms iteration = 6 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"1fa836d9-7854-4ada-8c89-c26d922646c6","tls_certs":["d7c3455b70a66c86ec2a7e597ca46fe0ea0c49dde5c1363f198c20aa77d58442"]}},"default_endpoint":{"silo_id":"1fa836d9-7854-4ada-8c89-c26d922646c6","tls_certs":["d7c3455b70a66c86ec2a7e597ca46fe0ea0c49dde5c1363f198c20aa77d58442"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
7882025-07-17T11:30:49.975ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7892025-07-17T11:30:49.976ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7902025-07-17T11:30:49.981ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 0ae78f24-48db-4cf7-91cb-8d5311f2a17f, lookup_type: ById(0ae78f24-48db-4cf7-91cb-8d5311f2a17f) } result = Ok(())
7912025-07-17T11:30:49.981ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
7922025-07-17T11:30:49.981ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
7932025-07-17T11:30:49.982ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7942025-07-17T11:30:49.982ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
7952025-07-17T11:30:49.982ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7962025-07-17T11:30:49.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7972025-07-17T11:30:49.984ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 219.264571ms iteration = 11 status = 0
7982025-07-17T11:30:49.985ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 220.149282ms iteration = 31 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
7992025-07-17T11:30:49.987ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
8002025-07-17T11:30:49.987ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
8012025-07-17T11:30:49.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
8022025-07-17T11:30:49.989ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8032025-07-17T11:30:49.989ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8042025-07-17T11:30:49.998ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8052025-07-17T11:30:49.998ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8062025-07-17T11:30:50.002ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8072025-07-17T11:30:50.002ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8082025-07-17T11:30:50.006ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8092025-07-17T11:30:50.007ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8102025-07-17T11:30:50.008ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 244.112548ms iteration = 11 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
8112025-07-17T11:30:50.009ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8122025-07-17T11:30:50.009ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8132025-07-17T11:30:50.011ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8142025-07-17T11:30:50.012ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8152025-07-17T11:30:50.014ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8162025-07-17T11:30:50.020ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
8172025-07-17T11:30:50.020ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8182025-07-17T11:30:50.021ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8192025-07-17T11:30:50.023ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8202025-07-17T11:30:50.023ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8212025-07-17T11:30:50.024ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8222025-07-17T11:30:50.025ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8232025-07-17T11:30:50.025ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8242025-07-17T11:30:50.026ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:550 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
8252025-07-17T11:30:50.026ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 261.286435ms iteration = 11 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
8262025-07-17T11:30:50.027ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8272025-07-17T11:30:50.028ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
8282025-07-17T11:30:50.028ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
8292025-07-17T11:30:50.028ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
8302025-07-17T11:30:50.032ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 268.297157ms iteration = 11 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
8312025-07-17T11:30:50.033ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
8322025-07-17T11:30:50.033ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
8332025-07-17T11:30:50.033ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 268.380748ms iteration = 11 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
8342025-07-17T11:30:50.034ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
8352025-07-17T11:30:50.034ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8362025-07-17T11:30:50.034ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8372025-07-17T11:30:50.040ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
8382025-07-17T11:30:50.040ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
8392025-07-17T11:30:50.041ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8402025-07-17T11:30:50.041ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8412025-07-17T11:30:50.042ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8422025-07-17T11:30:50.043ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
8432025-07-17T11:30:50.043ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8442025-07-17T11:30:50.044ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8452025-07-17T11:30:50.044ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
8462025-07-17T11:30:50.044ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
8472025-07-17T11:30:50.045ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
8482025-07-17T11:30:50.045ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
8492025-07-17T11:30:50.046ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
8502025-07-17T11:30:50.046ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8512025-07-17T11:30:50.046ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8522025-07-17T11:30:50.047ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8532025-07-17T11:30:50.049ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
8542025-07-17T11:30:50.049ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
8552025-07-17T11:30:50.049ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8562025-07-17T11:30:50.050ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
8572025-07-17T11:30:50.050ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8582025-07-17T11:30:50.051ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
8592025-07-17T11:30:50.051ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
8602025-07-17T11:30:50.053ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8612025-07-17T11:30:50.053ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8622025-07-17T11:30:50.053ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8632025-07-17T11:30:50.055ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8642025-07-17T11:30:50.055ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8652025-07-17T11:30:50.056ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
8662025-07-17T11:30:50.056ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
8672025-07-17T11:30:50.056ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
8682025-07-17T11:30:50.067ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
8692025-07-17T11:30:50.068ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
8702025-07-17T11:30:50.071ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8712025-07-17T11:30:50.072ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
8722025-07-17T11:30:50.072ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 4a9e4c5e-728e-4850-bd58-14548c02bf6f dns_zone_name = oxide.test found_so_far = 0 version = 2
8732025-07-17T11:30:50.072ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
8742025-07-17T11:30:50.072ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
8752025-07-17T11:30:50.072ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
8762025-07-17T11:30:50.072ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
8772025-07-17T11:30:50.074ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
8782025-07-17T11:30:50.074ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
8792025-07-17T11:30:50.074ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
8802025-07-17T11:30:50.075ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
8812025-07-17T11:30:50.075ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
8822025-07-17T11:30:50.078ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8832025-07-17T11:30:50.078ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8842025-07-17T11:30:50.082ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
8852025-07-17T11:30:50.082ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
8862025-07-17T11:30:50.082ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
8872025-07-17T11:30:50.083ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
8882025-07-17T11:30:50.083ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
8892025-07-17T11:30:50.083ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
8902025-07-17T11:30:50.084ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
8912025-07-17T11:30:50.084ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
8922025-07-17T11:30:50.084ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
8932025-07-17T11:30:50.084ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
8942025-07-17T11:30:50.086ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
8952025-07-17T11:30:50.088ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8962025-07-17T11:30:50.089ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 413.366409ms iteration = 11 status = null
8972025-07-17T11:30:50.089ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
8982025-07-17T11:30:50.089ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/all-sp-ids
8992025-07-17T11:30:50.090ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/all-sp-ids", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8250402-4d99-4836-a520-4c6349766aa7", "content-length": "78", "date": "Thu, 17 Jul 2025 11:30:49 GMT"} })
9002025-07-17T11:30:50.090ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = switch
9012025-07-17T11:30:50.090ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 1 sp_type = sled
9022025-07-17T11:30:50.090ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = sled
9032025-07-17T11:30:50.090ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
9042025-07-17T11:30:50.090ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
9052025-07-17T11:30:50.090ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
9062025-07-17T11:30:50.091ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = switch
9072025-07-17T11:30:50.091ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = sled
9082025-07-17T11:30:50.091ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 1 sp_type = sled
9092025-07-17T11:30:50.091ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9102025-07-17T11:30:50.091ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
9112025-07-17T11:30:50.093ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 0 sp_type = switch
9122025-07-17T11:30:50.093ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 4a9e4c5e-728e-4850-bd58-14548c02bf6f dns_zone_name = oxide.test found_so_far = 4 version = 2
9132025-07-17T11:30:50.093ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
9142025-07-17T11:30:50.093ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-07-17 11:26:35.847594 UTC dns_group = external generation = 2
9152025-07-17T11:30:50.093ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 417.627296ms iteration = 6 status = {"generation":2}
9162025-07-17T11:30:50.093ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
9172025-07-17T11:30:50.094ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 0 sp_type = sled
9182025-07-17T11:30:50.094ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
9192025-07-17T11:30:50.094ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
9202025-07-17T11:30:50.094ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
9212025-07-17T11:30:50.095ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
9222025-07-17T11:30:50.097ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 0 sp_type = switch
9232025-07-17T11:30:50.097ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = switch
9242025-07-17T11:30:50.097ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = switch
9252025-07-17T11:30:50.097ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9262025-07-17T11:30:50.097ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = tuf_artifact_replication body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/artifacts-config
9272025-07-17T11:30:50.098ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 0 sp_type = sled
9282025-07-17T11:30:50.098ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = sled
9292025-07-17T11:30:50.098ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = sled
9302025-07-17T11:30:50.102ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
9312025-07-17T11:30:50.102ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
9322025-07-17T11:30:50.102ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
9332025-07-17T11:30:50.105ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = tuf_artifact_replication result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/artifacts-config", status: 204, headers: {"x-request-id": "ffe49cc2-54ff-408c-94c1-80725bf79bad", "date": "Thu, 17 Jul 2025 11:30:50 GMT"} })
9342025-07-17T11:30:50.105ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
9352025-07-17T11:30:50.106ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
9362025-07-17T11:30:50.108ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9372025-07-17T11:30:50.108ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = tuf_artifact_replication body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/artifacts
9382025-07-17T11:30:50.108ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = tuf_artifact_replication result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/artifacts", status: 200, headers: {"content-type": "application/json", "x-request-id": "080dd972-c0aa-45f4-9342-2b52a7c99e01", "content-length": "26", "date": "Thu, 17 Jul 2025 11:30:50 GMT"} })
9392025-07-17T11:30:50.111ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
9402025-07-17T11:30:50.111ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
9412025-07-17T11:30:50.111ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
9422025-07-17T11:30:50.114ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
9432025-07-17T11:30:50.114ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
9442025-07-17T11:30:50.116ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9452025-07-17T11:30:50.118ZINFOnexus (BackgroundTasks): Successfully got artifact list background_task = tuf_artifact_replication file = nexus/src/app/background/tasks/tuf_artifact_replication.rs:672 sled = http://[fd00:1122:3344:101::1]:12345
9462025-07-17T11:30:50.119ZDEBGnexus (BackgroundTasks): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
9472025-07-17T11:30:50.119ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
9482025-07-17T11:30:50.119ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
9492025-07-17T11:30:50.121ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
9502025-07-17T11:30:50.121ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
9512025-07-17T11:30:50.122ZDEBGnexus (BackgroundTasks): activation complete background_task = tuf_artifact_replication elapsed = 430.711118ms iteration = 2 status = {"generation":1,"last_run_counters":{"copy_err":0,"copy_ok":0,"list_err":0,"list_ok":1,"put_config_err":0,"put_config_ok":1,"put_err":0,"put_ok":0},"lifetime_counters":{"copy_err":0,"copy_ok":0,"list_err":0,"list_ok":1,"put_config_err":0,"put_config_ok":1,"put_err":0,"put_ok":0},"local_repos":0,"request_debug_ringbuf":[{"generation":1,"operation":"put_config","target_sled":"0ae78f24-48db-4cf7-91cb-8d5311f2a17f","time":"2025-07-17T11:30:50.105482430Z"},{"operation":"list","target_sled":"0ae78f24-48db-4cf7-91cb-8d5311f2a17f","time":"2025-07-17T11:30:50.118168848Z"}]}
9522025-07-17T11:30:50.123ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9532025-07-17T11:30:50.125ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:372
9542025-07-17T11:30:50.125ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 360.204949ms iteration = 11 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
9552025-07-17T11:30:50.125ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
9562025-07-17T11:30:50.126ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
9572025-07-17T11:30:50.126ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1194 zone_address = fd00:1122:3344:101::2
9582025-07-17T11:30:50.126ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
9592025-07-17T11:30:50.127ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ac279dc-2a0d-4f93-8efd-e5e2786b927b", "content-length": "26", "date": "Thu, 17 Jul 2025 11:30:49 GMT"} })
9602025-07-17T11:30:50.127ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1197 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
9612025-07-17T11:30:50.127ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
9622025-07-17T11:30:50.127ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
9632025-07-17T11:30:50.127ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
9642025-07-17T11:30:50.129ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
9652025-07-17T11:30:50.129ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9662025-07-17T11:30:50.130ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9672025-07-17T11:30:50.131ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9682025-07-17T11:30:50.132ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
9692025-07-17T11:30:50.132ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9702025-07-17T11:30:50.132ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
9712025-07-17T11:30:50.133ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
9722025-07-17T11:30:50.135ZINFOnexus (BackgroundTasks): ensured all Crucible datasets present in inventory have database records background_task = blueprint_rendezvous file = nexus/reconfigurator/rendezvous/src/crucible_dataset.rs:113 num_already_exist = 9 num_inserted = 0 num_not_in_inventory = 0
9732025-07-17T11:30:50.138ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 450.988525ms iteration = 3 status = {"blueprint_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","inventory_collection_id":"bd15575d-5499-4736-988b-3d1df00d704b","stats":{"crucible_dataset":{"num_already_exist":9,"num_inserted":0,"num_not_in_inventory":0},"debug_dataset":{"num_already_exist":8,"num_already_tombstoned":0,"num_inserted":1,"num_not_in_inventory":0,"num_tombstoned":0}}}
9742025-07-17T11:30:50.138ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
9752025-07-17T11:30:50.138ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9762025-07-17T11:30:50.138ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9772025-07-17T11:30:50.138ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9782025-07-17T11:30:50.138ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9792025-07-17T11:30:50.140ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9802025-07-17T11:30:50.140ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9812025-07-17T11:30:50.142ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 1 sp_type = sled
9822025-07-17T11:30:50.142ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9832025-07-17T11:30:50.145ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 1 sp_type = sled
9842025-07-17T11:30:50.145ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 1 sp_type = sled
9852025-07-17T11:30:50.146ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 1 sp_type = sled
9862025-07-17T11:30:50.146ZINFOnexus (BackgroundTasks): VPC route manager sled 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
9872025-07-17T11:30:50.146ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
9882025-07-17T11:30:50.146ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d187659-2141-4542-a773-a1beedb10c4d", "content-length": "372", "date": "Thu, 17 Jul 2025 11:30:50 GMT"} })
9892025-07-17T11:30:50.146ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
9902025-07-17T11:30:50.147ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
9912025-07-17T11:30:50.148ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
9922025-07-17T11:30:50.148ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9932025-07-17T11:30:50.149ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9942025-07-17T11:30:50.155ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 0 sp_type = switch start_ena = None
9952025-07-17T11:30:50.155ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
9962025-07-17T11:30:50.156ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "3068c668-2b4e-4c2b-bff0-c9b54420995c", "content-length": "81", "date": "Thu, 17 Jul 2025 11:30:49 GMT"} })
9972025-07-17T11:30:50.156ZDEBGnexus (service-firewall-rule-progator): resolved names for firewall rules background_task = service_firewall_rule_propagation instance_interfaces = {} subnet_interfaces = {Name("nexus"): [NetworkInterface { id: 1c4a40e7-c575-4aa4-8a05-4f7305d5faa6, kind: Service { id: 5e9f3641-e36d-4c7a-ab2f-ad78baee50f9 }, name: Name("nexus-5e9f3641-e36d-4c7a-ab2f-ad78baee50f9"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 225, 75])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 26ccceb4-722f-401c-9efa-2678860d14f3, kind: Service { id: f39d3e70-66e4-468a-a44d-746d22bea3a0 }, name: Name("nexus-f39d3e70-66e4-468a-a44d-746d22bea3a0"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 232, 228])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 89aa3989-5913-43b2-86dd-df3c95477576, kind: Service { id: 1017ad63-50a5-4825-ad3a-e3d3e6cffc2e }, name: Name("nexus-1017ad63-50a5-4825-ad3a-e3d3e6cffc2e"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 202, 53])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], Name("external-dns"): [NetworkInterface { id: 82afb5bb-4e12-4b36-a03e-f32580a36236, kind: Service { id: 79e5c77d-18b3-4951-8d11-cfa3eae2b3e3 }, name: Name("external-dns-79e5c77d-18b3-4951-8d11-cfa3eae2b3e3"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 136, 29])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 949f6c29-3095-471f-b7dd-aa2aa681af87, kind: Service { id: 3dd3bf3b-7760-4750-b1d3-52ee777335f0 }, name: Name("external-dns-3dd3bf3b-7760-4750-b1d3-52ee777335f0"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 144, 121])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }]} subnet_networks = {Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })], Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })]} vpc_interfaces = {}
9982025-07-17T11:30:50.156ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9992025-07-17T11:30:50.157ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
10002025-07-17T11:30:50.159ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10012025-07-17T11:30:50.161ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 0 sp_type = sled start_ena = None
10022025-07-17T11:30:50.161ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
10032025-07-17T11:30:50.161ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0b8f81b-fa1b-442a-84ff-f77f4a5b618f", "content-length": "81", "date": "Thu, 17 Jul 2025 11:30:49 GMT"} })
10042025-07-17T11:30:50.162ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Fleet result = Ok(())
10052025-07-17T11:30:50.162ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
10062025-07-17T11:30:50.162ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
10072025-07-17T11:30:50.164ZDEBGnexus (service-firewall-rule-progator): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified. background_task = service_firewall_rule_propagation
10082025-07-17T11:30:50.168ZDEBGnexus (service-firewall-rule-progator): resolved firewall rules for sled agents background_task = service_firewall_rule_propagation sled_agent_rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 82afb5bb-4e12-4b36-a03e-f32580a36236, kind: Service { id: 79e5c77d-18b3-4951-8d11-cfa3eae2b3e3 }, name: Name("external-dns-79e5c77d-18b3-4951-8d11-cfa3eae2b3e3"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 136, 29])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 949f6c29-3095-471f-b7dd-aa2aa681af87, kind: Service { id: 3dd3bf3b-7760-4750-b1d3-52ee777335f0 }, name: Name("external-dns-3dd3bf3b-7760-4750-b1d3-52ee777335f0"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 144, 121])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 1c4a40e7-c575-4aa4-8a05-4f7305d5faa6, kind: Service { id: 5e9f3641-e36d-4c7a-ab2f-ad78baee50f9 }, name: Name("nexus-5e9f3641-e36d-4c7a-ab2f-ad78baee50f9"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 225, 75])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 26ccceb4-722f-401c-9efa-2678860d14f3, kind: Service { id: f39d3e70-66e4-468a-a44d-746d22bea3a0 }, name: Name("nexus-f39d3e70-66e4-468a-a44d-746d22bea3a0"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 232, 228])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 89aa3989-5913-43b2-86dd-df3c95477576, kind: Service { id: 1017ad63-50a5-4825-ad3a-e3d3e6cffc2e }, name: Name("nexus-1017ad63-50a5-4825-ad3a-e3d3e6cffc2e"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 202, 53])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 82afb5bb-4e12-4b36-a03e-f32580a36236, kind: Service { id: 79e5c77d-18b3-4951-8d11-cfa3eae2b3e3 }, name: Name("external-dns-79e5c77d-18b3-4951-8d11-cfa3eae2b3e3"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 136, 29])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 949f6c29-3095-471f-b7dd-aa2aa681af87, kind: Service { id: 3dd3bf3b-7760-4750-b1d3-52ee777335f0 }, name: Name("external-dns-3dd3bf3b-7760-4750-b1d3-52ee777335f0"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 144, 121])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp(Some(VpcFirewallIcmpFilter { icmp_type: 3, code: Some(IcmpParamRange { first: 3, last: 4 }) })), Icmp(Some(VpcFirewallIcmpFilter { icmp_type: 5, code: None })), Icmp(Some(VpcFirewallIcmpFilter { icmp_type: 11, code: None }))]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 1c4a40e7-c575-4aa4-8a05-4f7305d5faa6, kind: Service { id: 5e9f3641-e36d-4c7a-ab2f-ad78baee50f9 }, name: Name("nexus-5e9f3641-e36d-4c7a-ab2f-ad78baee50f9"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 225, 75])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 26ccceb4-722f-401c-9efa-2678860d14f3, kind: Service { id: f39d3e70-66e4-468a-a44d-746d22bea3a0 }, name: Name("nexus-f39d3e70-66e4-468a-a44d-746d22bea3a0"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 232, 228])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 89aa3989-5913-43b2-86dd-df3c95477576, kind: Service { id: 1017ad63-50a5-4825-ad3a-e3d3e6cffc2e }, name: Name("nexus-1017ad63-50a5-4825-ad3a-e3d3e6cffc2e"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 202, 53])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }]
10092025-07-17T11:30:50.168ZDEBGnexus (service-firewall-rule-progator): resolved 3 rules for sleds background_task = service_firewall_rule_propagation
10102025-07-17T11:30:50.171ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1194 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 zone_address = fd00:1122:3344:101::2
10112025-07-17T11:30:50.171ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
10122025-07-17T11:30:50.171ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "e65aa67f-37b4-413e-8c39-ed91cb22fadc", "content-length": "26", "date": "Thu, 17 Jul 2025 11:30:49 GMT"} })
10132025-07-17T11:30:50.171ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1197 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
10142025-07-17T11:30:50.172ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
10152025-07-17T11:30:50.172ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
10162025-07-17T11:30:50.173ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "3f6861ab-c629-4186-b18d-86e0b80eba5b", "date": "Thu, 17 Jul 2025 11:30:50 GMT"} })
10172025-07-17T11:30:50.173ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
10182025-07-17T11:30:50.173ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
10192025-07-17T11:30:50.173ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
10202025-07-17T11:30:50.175ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
10212025-07-17T11:30:50.176ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
10222025-07-17T11:30:50.179ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
10232025-07-17T11:30:50.179ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
10242025-07-17T11:30:50.183ZINFOnexus (BackgroundTasks): VPC route manager sled 0ae78f24-48db-4cf7-91cb-8d5311f2a17f push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
10252025-07-17T11:30:50.183ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
10262025-07-17T11:30:50.183ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "97cc0938-2a55-44b2-99c1-d0dff3e73cd3", "date": "Thu, 17 Jul 2025 11:30:50 GMT"} })
10272025-07-17T11:30:50.185ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 420.782357ms iteration = 11 status = {}
10282025-07-17T11:30:50.195ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 1 sp_type = sled start_ena = None
10292025-07-17T11:30:50.195ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/1/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
10302025-07-17T11:30:50.196ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/1/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c1bacdf-e374-4c33-bd3e-e1a70d2f0556", "content-length": "81", "date": "Thu, 17 Jul 2025 11:30:49 GMT"} })
10312025-07-17T11:30:50.198ZDEBGnexus (BackgroundTasks): activation complete background_task = sp_ereport_ingester elapsed = 433.264803ms iteration = 11 status = {"errors":[],"sps":[]}
10322025-07-17T11:30:50.263ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
10332025-07-17T11:30:50.274ZINFOnexus (BackgroundTasks): instance reincarnation completed successfully background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:113 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0
10342025-07-17T11:30:50.274ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 589.710645ms iteration = 6 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
10352025-07-17T11:30:50.277ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
10362025-07-17T11:30:50.277ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
10372025-07-17T11:30:50.280ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10382025-07-17T11:30:50.283ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
10392025-07-17T11:30:50.283ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
10402025-07-17T11:30:50.283ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
10412025-07-17T11:30:50.325ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
10422025-07-17T11:30:50.325ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "cef04163-438b-4838-ab7e-07c4b647cfc6", "date": "Thu, 17 Jul 2025 11:30:50 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
10432025-07-17T11:30:50.325ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
10442025-07-17T11:30:50.328ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "af4c8c22-d217-4716-b1ec-b49b0eb94e4a", "date": "Thu, 17 Jul 2025 11:30:50 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
10452025-07-17T11:30:50.328ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
10462025-07-17T11:30:50.330ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
10472025-07-17T11:30:50.330ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 654.632421ms iteration = 8 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
10482025-07-17T11:30:50.339ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 606.546427ms iteration = 11 status = {}
10492025-07-17T11:30:50.341ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10502025-07-17T11:30:50.343ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
10512025-07-17T11:30:50.343ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
10522025-07-17T11:30:50.344ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
10532025-07-17T11:30:50.361ZDEBGnexus (service-firewall-rule-progator): resolved sleds for vpc oxide-services background_task = service_firewall_rule_propagation vpc_to_sled = [Sled { identity: SledIdentity { id: 0ae78f24-48db-4cf7-91cb-8d5311f2a17f, time_created: 2025-07-17T11:26:26.884734Z, time_modified: 2025-07-17T11:26:26.884734Z }, time_deleted: None, rcgen: Generation(Generation(19)), rack_id: 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5, is_scrimlet: true, serial_number: "sock", part_number: "i86pc", revision: SqlU32(0), usable_hardware_threads: SqlU32(48), usable_physical_ram: ByteCount(ByteCount(102952026112)), reservoir_size: ByteCount(ByteCount(56094621696)), ip: fd00:1122:3344:101::1, port: SqlU16(12345), last_used_address: fd00:1122:3344:101::ffff, policy: InService, state: Active, sled_agent_gen: Generation(Generation(1)), repo_depot_port: SqlU16(12348) }]
10542025-07-17T11:30:50.361ZDEBGnexus (service-firewall-rule-progator): sending firewall rules to sled agents background_task = service_firewall_rule_propagation
10552025-07-17T11:30:50.363ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
10562025-07-17T11:30:50.364ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "bd0c84af-3b25-4d54-b819-dfbe59e4f7a1", "date": "Thu, 17 Jul 2025 11:30:50 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
10572025-07-17T11:30:50.364ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
10582025-07-17T11:30:50.365ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "d42d4bc6-0c0b-43e8-ad96-887a18726030", "date": "Thu, 17 Jul 2025 11:30:50 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
10592025-07-17T11:30:50.365ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
10602025-07-17T11:30:50.368ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "64953934-bb14-4c4e-be1f-1a26ef37db45", "date": "Thu, 17 Jul 2025 11:30:50 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
10612025-07-17T11:30:50.368ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
10622025-07-17T11:30:50.370ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
10632025-07-17T11:30:50.370ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 694.697959ms iteration = 8 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
10642025-07-17T11:30:50.430ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
10652025-07-17T11:30:50.434ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
10662025-07-17T11:30:50.435ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
10672025-07-17T11:30:50.437ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "2bd372b8-99dc-4bb7-a1c6-0b4896554af4", "content-length": "691", "date": "Thu, 17 Jul 2025 11:30:50 GMT"} })
10682025-07-17T11:30:50.437ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
10692025-07-17T11:30:50.437ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
10702025-07-17T11:30:50.437ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
10712025-07-17T11:30:50.438ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
10722025-07-17T11:30:50.438ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
10732025-07-17T11:30:50.439ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 674.410522ms iteration = 11 status = {}
10742025-07-17T11:30:50.440ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10752025-07-17T11:30:50.444ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Sled { parent: Fleet, key: 0ae78f24-48db-4cf7-91cb-8d5311f2a17f, lookup_type: ById(0ae78f24-48db-4cf7-91cb-8d5311f2a17f) } result = Ok(())
10762025-07-17T11:30:50.444ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
10772025-07-17T11:30:50.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
10782025-07-17T11:30:50.447ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 151caa7c-6a05-4995-94b4-7946d428117d, rack_id: 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5, switch_location: "switch0", port_name: Name(Name("qsfp0")), port_settings_id: Some(2e30c7ae-26da-46e4-9127-aac194921f20) } rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0
10792025-07-17T11:30:50.447ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
10802025-07-17T11:30:50.448ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
10812025-07-17T11:30:50.449ZDEBGnexus (service-firewall-rule-progator): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = service_firewall_rule_propagation body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules
10822025-07-17T11:30:50.455ZDEBGnexus (service-firewall-rule-progator): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = service_firewall_rule_propagation result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules", status: 204, headers: {"x-request-id": "66d273bb-14b6-40e4-a53e-dfeae74ddb21", "date": "Thu, 17 Jul 2025 11:30:50 GMT"} })
10832025-07-17T11:30:50.456ZINFOnexus (service-firewall-rule-progator): updated firewall rules on 1 sleds background_task = service_firewall_rule_propagation file = nexus/networking/src/firewall_rules.rs:491
10842025-07-17T11:30:50.456ZDEBGnexus (service-firewall-rule-progator): successfully propagated service firewall rules background_task = service_firewall_rule_propagation elapsed = 771.970197ms
10852025-07-17T11:30:50.456ZDEBGnexus (BackgroundTasks): activation complete background_task = service_firewall_rule_propagation elapsed = 771.978477ms iteration = 2 status = {}
10862025-07-17T11:30:50.461ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1832 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = Name(Name("qsfp0"))
10872025-07-17T11:30:50.461ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1859 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0 switch_port_id = Name(Name("qsfp0")) switch_port_settings_id = 2e30c7ae-26da-46e4-9127-aac194921f20
10882025-07-17T11:30:50.462ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
10892025-07-17T11:30:50.462ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
10902025-07-17T11:30:50.462ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
10912025-07-17T11:30:50.462ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
10922025-07-17T11:30:50.462ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
10932025-07-17T11:30:50.462ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2006 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
10942025-07-17T11:30:50.462ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
10952025-07-17T11:30:50.462ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2042 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
10962025-07-17T11:30:50.462ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
10972025-07-17T11:30:50.462ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
10982025-07-17T11:30:50.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
10992025-07-17T11:30:50.465ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0
11002025-07-17T11:30:50.465ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
11012025-07-17T11:30:50.624ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "2dff0ce7-ea75-4bc7-b9cd-8f06274a7bc3", "date": "Thu, 17 Jul 2025 11:30:50 GMT"} })
11022025-07-17T11:30:50.624ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch = Switch0
11032025-07-17T11:30:50.624ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
11042025-07-17T11:30:50.630ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "e052b7f4-94af-4be5-b529-605d5ee9af2e", "content-length": "682", "date": "Thu, 17 Jul 2025 11:30:50 GMT"} })
11052025-07-17T11:30:50.630ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
11062025-07-17T11:30:50.631ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
11072025-07-17T11:30:50.634ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
11082025-07-17T11:30:50.635ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
11092025-07-17T11:30:50.639ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
11102025-07-17T11:30:50.639ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
11112025-07-17T11:30:50.641ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
11122025-07-17T11:30:50.641ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
11132025-07-17T11:30:50.648ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 883.6297ms iteration = 11 status = {}
11142025-07-17T11:30:51.691ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 332 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = f8815ae7-68bb-4fca-b562-67ec4ac58745 response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
11152025-07-17T11:30:54.763ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 62 reason = Timeout
11162025-07-17T11:30:54.763ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
11172025-07-17T11:30:54.763ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
11182025-07-17T11:30:54.765ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11192025-07-17T11:30:54.768ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
11202025-07-17T11:30:54.768ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
11212025-07-17T11:30:54.768ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
11222025-07-17T11:30:54.770ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
11232025-07-17T11:30:54.770ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 6.933481ms iteration = 62 status = {"chicken_switches_updated":false}
11242025-07-17T11:30:59.763ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 32 reason = Timeout
11252025-07-17T11:30:59.763ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 32 reason = Timeout
11262025-07-17T11:30:59.763ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
11272025-07-17T11:30:59.763ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
11282025-07-17T11:30:59.763ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 63 reason = Timeout
11292025-07-17T11:30:59.763ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
11302025-07-17T11:30:59.763ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
11312025-07-17T11:30:59.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
11322025-07-17T11:30:59.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
11332025-07-17T11:30:59.765ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11342025-07-17T11:30:59.766ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11352025-07-17T11:30:59.768ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
11362025-07-17T11:30:59.768ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
11372025-07-17T11:30:59.768ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
11382025-07-17T11:30:59.769ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
11392025-07-17T11:30:59.769ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
11402025-07-17T11:30:59.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
11412025-07-17T11:30:59.769ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
11422025-07-17T11:30:59.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
11432025-07-17T11:30:59.770ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11442025-07-17T11:30:59.770ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
11452025-07-17T11:30:59.770ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 7.305266ms iteration = 63 status = {"chicken_switches_updated":false}
11462025-07-17T11:30:59.773ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
11472025-07-17T11:30:59.773ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
11482025-07-17T11:30:59.773ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
11492025-07-17T11:30:59.784ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
11502025-07-17T11:30:59.784ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.661423ms iteration = 32 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
11512025-07-17T11:30:59.798ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
11522025-07-17T11:30:59.798ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
11532025-07-17T11:30:59.833ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 70.199295ms iteration = 32 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
11542025-07-17T11:31:01.683ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62728 req_id = 4c61abd0-5504-4bfc-9deb-d59133d05b37 roles = RoleSet { roles: {} } uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?sort_by=id_ascending
11552025-07-17T11:31:01.683ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62728 req_id = 4c61abd0-5504-4bfc-9deb-d59133d05b37 resource = Database result = Ok(()) uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?sort_by=id_ascending
11562025-07-17T11:31:01.686ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 3293 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62728 req_id = 4c61abd0-5504-4bfc-9deb-d59133d05b37 response_code = 200 uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?sort_by=id_ascending
11572025-07-17T11:31:01.686ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62728 req_id = 234e885c-4339-4604-9fdd-7f4d10d642dc roles = RoleSet { roles: {} } uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmMzlkM2U3MC02NmU0LTQ2OGEtYTQ0ZC03NDZkMjJiZWEzYTAifX0%3D
11582025-07-17T11:31:01.687ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62728 req_id = 234e885c-4339-4604-9fdd-7f4d10d642dc resource = Database result = Ok(()) uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmMzlkM2U3MC02NmU0LTQ2OGEtYTQ0ZC03NDZkMjJiZWEzYTAifX0%3D
11592025-07-17T11:31:01.689ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 2651 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62728 req_id = 234e885c-4339-4604-9fdd-7f4d10d642dc response_code = 200 uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmMzlkM2U3MC02NmU0LTQ2OGEtYTQ0ZC03NDZkMjJiZWEzYTAifX0%3D
11602025-07-17T11:31:01.691ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 300 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = 70298fd9-d7d8-45a2-876b-28bbc3c0615f response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
11612025-07-17T11:31:04.764ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 64 reason = Timeout
11622025-07-17T11:31:04.764ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
11632025-07-17T11:31:04.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
11642025-07-17T11:31:04.765ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11652025-07-17T11:31:04.768ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
11662025-07-17T11:31:04.768ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
11672025-07-17T11:31:04.768ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
11682025-07-17T11:31:04.770ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
11692025-07-17T11:31:04.770ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 6.304543ms iteration = 64 status = {"chicken_switches_updated":false}
11702025-07-17T11:31:09.763ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 33 reason = Timeout
11712025-07-17T11:31:09.763ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
11722025-07-17T11:31:09.763ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 33 reason = Timeout
11732025-07-17T11:31:09.763ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 65 reason = Timeout
11742025-07-17T11:31:09.763ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
11752025-07-17T11:31:09.763ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
11762025-07-17T11:31:09.763ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
11772025-07-17T11:31:09.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
11782025-07-17T11:31:09.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
11792025-07-17T11:31:09.765ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11802025-07-17T11:31:09.765ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11812025-07-17T11:31:09.766ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
11822025-07-17T11:31:09.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
11832025-07-17T11:31:09.767ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.206186ms iteration = 33 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
11842025-07-17T11:31:09.768ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
11852025-07-17T11:31:09.768ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
11862025-07-17T11:31:09.768ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
11872025-07-17T11:31:09.768ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
11882025-07-17T11:31:09.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
11892025-07-17T11:31:09.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
11902025-07-17T11:31:09.770ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
11912025-07-17T11:31:09.770ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 7.211526ms iteration = 65 status = {"chicken_switches_updated":false}
11922025-07-17T11:31:09.771ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
11932025-07-17T11:31:09.771ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
11942025-07-17T11:31:09.772ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11952025-07-17T11:31:09.775ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
11962025-07-17T11:31:09.775ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
11972025-07-17T11:31:09.775ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
11982025-07-17T11:31:09.784ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
11992025-07-17T11:31:09.784ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.046957ms iteration = 33 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
12002025-07-17T11:31:11.692ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 596 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = 773635e4-b6ec-4c5a-a960-1b983a10ea8f response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
12012025-07-17T11:31:14.763ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 66 reason = Timeout
12022025-07-17T11:31:14.763ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
12032025-07-17T11:31:14.763ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
12042025-07-17T11:31:14.765ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12052025-07-17T11:31:14.768ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
12062025-07-17T11:31:14.768ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
12072025-07-17T11:31:14.768ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
12082025-07-17T11:31:14.769ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
12092025-07-17T11:31:14.769ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 6.263803ms iteration = 66 status = {"chicken_switches_updated":false}
12102025-07-17T11:31:16.683ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62728 req_id = 489accf8-e1dc-4707-b060-c30e4f7be3a6 roles = RoleSet { roles: {} } uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?sort_by=id_ascending
12112025-07-17T11:31:16.684ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62728 req_id = 489accf8-e1dc-4707-b060-c30e4f7be3a6 resource = Database result = Ok(()) uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?sort_by=id_ascending
12122025-07-17T11:31:16.685ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 2138 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62728 req_id = 489accf8-e1dc-4707-b060-c30e4f7be3a6 response_code = 200 uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?sort_by=id_ascending
12132025-07-17T11:31:16.686ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62728 req_id = da79d5be-0332-4cfe-97fd-f40ed7f2b66b roles = RoleSet { roles: {} } uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmMzlkM2U3MC02NmU0LTQ2OGEtYTQ0ZC03NDZkMjJiZWEzYTAifX0%3D
12142025-07-17T11:31:16.686ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62728 req_id = da79d5be-0332-4cfe-97fd-f40ed7f2b66b resource = Database result = Ok(()) uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmMzlkM2U3MC02NmU0LTQ2OGEtYTQ0ZC03NDZkMjJiZWEzYTAifX0%3D
12152025-07-17T11:31:16.688ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 2172 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:62728 req_id = da79d5be-0332-4cfe-97fd-f40ed7f2b66b response_code = 200 uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmMzlkM2U3MC02NmU0LTQ2OGEtYTQ0ZC03NDZkMjJiZWEzYTAifX0%3D
12162025-07-17T11:31:19.675ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 12 reason = Timeout
12172025-07-17T11:31:19.675ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
12182025-07-17T11:31:19.675ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
12192025-07-17T11:31:19.678ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3673, tv_nsec: 431383554 } })
12202025-07-17T11:31:19.678ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1186
12212025-07-17T11:31:19.708ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1194 zone_address = fd00:1122:3344:101::2
12222025-07-17T11:31:19.708ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
12232025-07-17T11:31:19.708ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ebb7263-8fda-4a1f-8dfb-93ec5d27f34e", "content-length": "26", "date": "Thu, 17 Jul 2025 11:31:19 GMT"} })
12242025-07-17T11:31:19.708ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1197 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
12252025-07-17T11:31:19.709ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
12262025-07-17T11:31:19.732ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 12 reason = Timeout
12272025-07-17T11:31:19.732ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 12 reason = Timeout
12282025-07-17T11:31:19.732ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
12292025-07-17T11:31:19.732ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
12302025-07-17T11:31:19.734ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3673, tv_nsec: 487998010 } })
12312025-07-17T11:31:19.735ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1186
12322025-07-17T11:31:19.736ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.55204ms iteration = 12 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
12332025-07-17T11:31:19.746ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
12342025-07-17T11:31:19.746ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
12352025-07-17T11:31:19.752ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 77.549873ms iteration = 12 status = null
12362025-07-17T11:31:19.763ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 12 reason = Timeout
12372025-07-17T11:31:19.763ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
12382025-07-17T11:31:19.763ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 12 reason = Timeout
12392025-07-17T11:31:19.763ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
12402025-07-17T11:31:19.763ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 12 reason = Timeout
12412025-07-17T11:31:19.763ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
12422025-07-17T11:31:19.763ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 12 reason = Timeout
12432025-07-17T11:31:19.763ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
12442025-07-17T11:31:19.763ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
12452025-07-17T11:31:19.763ZDEBGnexus (BackgroundTasks): activating background_task = sp_ereport_ingester iteration = 12 reason = Timeout
12462025-07-17T11:31:19.763ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
12472025-07-17T11:31:19.763ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 34 reason = Timeout
12482025-07-17T11:31:19.763ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
12492025-07-17T11:31:19.763ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 13 reason = Timeout
12502025-07-17T11:31:19.763ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
12512025-07-17T11:31:19.763ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 12 reason = Timeout
12522025-07-17T11:31:19.763ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 12 reason = Timeout
12532025-07-17T11:31:19.763ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
12542025-07-17T11:31:19.763ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 12 reason = Timeout
12552025-07-17T11:31:19.763ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
12562025-07-17T11:31:19.763ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
12572025-07-17T11:31:19.763ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 12 reason = Timeout
12582025-07-17T11:31:19.763ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
12592025-07-17T11:31:19.763ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 34 reason = Timeout
12602025-07-17T11:31:19.763ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12612025-07-17T11:31:19.763ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 67 reason = Timeout
12622025-07-17T11:31:19.763ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
12632025-07-17T11:31:19.763ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 12 reason = Timeout
12642025-07-17T11:31:19.763ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 12 reason = Timeout
12652025-07-17T11:31:19.763ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
12662025-07-17T11:31:19.763ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
12672025-07-17T11:31:19.763ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 12 reason = Timeout
12682025-07-17T11:31:19.763ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 12 reason = Timeout
12692025-07-17T11:31:19.763ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
12702025-07-17T11:31:19.763ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
12712025-07-17T11:31:19.763ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 12 reason = Timeout
12722025-07-17T11:31:19.763ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
12732025-07-17T11:31:19.763ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
12742025-07-17T11:31:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
12752025-07-17T11:31:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
12762025-07-17T11:31:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
12772025-07-17T11:31:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
12782025-07-17T11:31:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
12792025-07-17T11:31:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12802025-07-17T11:31:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
12812025-07-17T11:31:19.764ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12225, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3673, tv_nsec: 517364367 } })
12822025-07-17T11:31:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
12832025-07-17T11:31:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
12842025-07-17T11:31:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
12852025-07-17T11:31:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
12862025-07-17T11:31:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
12872025-07-17T11:31:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
12882025-07-17T11:31:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
12892025-07-17T11:31:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
12902025-07-17T11:31:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
12912025-07-17T11:31:19.768ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 4.468348ms iteration = 12 status = {"errors":[],"garbage_collect_requested":[]}
12922025-07-17T11:31:19.772ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
12932025-07-17T11:31:19.773ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
12942025-07-17T11:31:19.773ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12952025-07-17T11:31:19.774ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12962025-07-17T11:31:19.774ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12972025-07-17T11:31:19.775ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12982025-07-17T11:31:19.775ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
12992025-07-17T11:31:19.775ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
13002025-07-17T11:31:19.776ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
13012025-07-17T11:31:19.776ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13022025-07-17T11:31:19.776ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13032025-07-17T11:31:19.777ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13042025-07-17T11:31:19.778ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13052025-07-17T11:31:19.779ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
13062025-07-17T11:31:19.779ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
13072025-07-17T11:31:19.779ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 16.020911ms iteration = 34 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
13082025-07-17T11:31:19.779ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
13092025-07-17T11:31:19.780ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13102025-07-17T11:31:19.780ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
13112025-07-17T11:31:19.780ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
13122025-07-17T11:31:19.781ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
13132025-07-17T11:31:19.781ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
13142025-07-17T11:31:19.781ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
13152025-07-17T11:31:19.781ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
13162025-07-17T11:31:19.781ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
13172025-07-17T11:31:19.781ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
13182025-07-17T11:31:19.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
13192025-07-17T11:31:19.782ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
13202025-07-17T11:31:19.782ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
13212025-07-17T11:31:19.783ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
13222025-07-17T11:31:19.783ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
13232025-07-17T11:31:19.783ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
13242025-07-17T11:31:19.783ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
13252025-07-17T11:31:19.784ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
13262025-07-17T11:31:19.784ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
13272025-07-17T11:31:19.784ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
13282025-07-17T11:31:19.784ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
13292025-07-17T11:31:19.786ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
13302025-07-17T11:31:19.786ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
13312025-07-17T11:31:19.790ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
13322025-07-17T11:31:19.790ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
13332025-07-17T11:31:19.790ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
13342025-07-17T11:31:19.790ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
13352025-07-17T11:31:19.790ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 27.038517ms iteration = 67 status = {"chicken_switches_updated":false}
13362025-07-17T11:31:19.790ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
13372025-07-17T11:31:19.792ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1194 zone_address = fd00:1122:3344:101::2
13382025-07-17T11:31:19.792ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
13392025-07-17T11:31:19.792ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
13402025-07-17T11:31:19.792ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13412025-07-17T11:31:19.792ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "089c5a97-08f8-4f0f-a337-ee2170ebe933", "content-length": "26", "date": "Thu, 17 Jul 2025 11:31:19 GMT"} })
13422025-07-17T11:31:19.792ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1197 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
13432025-07-17T11:31:19.793ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3673, tv_nsec: 546248388 } })
13442025-07-17T11:31:19.793ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
13452025-07-17T11:31:19.793ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1186 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
13462025-07-17T11:31:19.795ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
13472025-07-17T11:31:19.796ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
13482025-07-17T11:31:19.796ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
13492025-07-17T11:31:19.796ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
13502025-07-17T11:31:19.839ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
13512025-07-17T11:31:19.839ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
13522025-07-17T11:31:19.852ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
13532025-07-17T11:31:19.853ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
13542025-07-17T11:31:19.855ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 92.229556ms iteration = 12 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
13552025-07-17T11:31:19.856ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 93.150278ms iteration = 12 status = {"errors":[],"finish_invoked_ok":[]}
13562025-07-17T11:31:19.858ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/all-sp-ids
13572025-07-17T11:31:19.859ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/all-sp-ids", status: 200, headers: {"content-type": "application/json", "x-request-id": "df777ea9-2963-4fe9-9b40-1c60a6c7395b", "content-length": "78", "date": "Thu, 17 Jul 2025 11:31:19 GMT"} })
13582025-07-17T11:31:19.859ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = switch
13592025-07-17T11:31:19.859ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 1 sp_type = sled
13602025-07-17T11:31:19.859ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = sled
13612025-07-17T11:31:19.859ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
13622025-07-17T11:31:19.859ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
13632025-07-17T11:31:19.859ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
13642025-07-17T11:31:19.860ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = switch
13652025-07-17T11:31:19.860ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
13662025-07-17T11:31:19.860ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 1 sp_type = sled
13672025-07-17T11:31:19.860ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = sled
13682025-07-17T11:31:19.863ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 1 sp_type = sled
13692025-07-17T11:31:19.865ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
13702025-07-17T11:31:19.865ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
13712025-07-17T11:31:19.867ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13722025-07-17T11:31:19.871ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 1 sp_type = sled
13732025-07-17T11:31:19.871ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 1 sp_type = sled
13742025-07-17T11:31:19.871ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 1 sp_type = sled
13752025-07-17T11:31:19.874ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
13762025-07-17T11:31:19.874ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
13772025-07-17T11:31:19.874ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 0 sp_type = sled
13782025-07-17T11:31:19.874ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 1 sp_type = sled start_ena = None
13792025-07-17T11:31:19.874ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/1/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
13802025-07-17T11:31:19.874ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/1/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb37c90a-a33c-4b57-a85d-efe2e516422d", "content-length": "81", "date": "Thu, 17 Jul 2025 11:31:19 GMT"} })
13812025-07-17T11:31:19.874ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
13822025-07-17T11:31:19.875ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 112.147458ms iteration = 12 status = {"errors":[],"requests_created_ok":[]}
13832025-07-17T11:31:19.876ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:550 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
13842025-07-17T11:31:19.876ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
13852025-07-17T11:31:19.876ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 113.260703ms iteration = 12 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
13862025-07-17T11:31:19.877ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
13872025-07-17T11:31:19.877ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 0 sp_type = sled
13882025-07-17T11:31:19.877ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = sled
13892025-07-17T11:31:19.878ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = sled
13902025-07-17T11:31:19.881ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13912025-07-17T11:31:19.881ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 117.60747ms iteration = 12 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
13922025-07-17T11:31:19.882ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1194 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 zone_address = fd00:1122:3344:101::2
13932025-07-17T11:31:19.882ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
13942025-07-17T11:31:19.883ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b932eb0-38b9-434c-8749-d614dd90ffdc", "content-length": "26", "date": "Thu, 17 Jul 2025 11:31:19 GMT"} })
13952025-07-17T11:31:19.883ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1197 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
13962025-07-17T11:31:19.883ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 0 sp_type = sled start_ena = None
13972025-07-17T11:31:19.883ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
13982025-07-17T11:31:19.884ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "b18c3c18-3b15-4dd8-9de8-c3820ee5ada9", "content-length": "81", "date": "Thu, 17 Jul 2025 11:31:19 GMT"} })
13992025-07-17T11:31:19.884ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
14002025-07-17T11:31:19.887ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
14012025-07-17T11:31:19.887ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
14022025-07-17T11:31:19.887ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
14032025-07-17T11:31:19.899ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
14042025-07-17T11:31:19.900ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
14052025-07-17T11:31:19.901ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 169.014139ms iteration = 12 status = {}
14062025-07-17T11:31:19.907ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
14072025-07-17T11:31:19.907ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
14082025-07-17T11:31:19.909ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14092025-07-17T11:31:19.914ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 0ae78f24-48db-4cf7-91cb-8d5311f2a17f, lookup_type: ById(0ae78f24-48db-4cf7-91cb-8d5311f2a17f) } result = Ok(())
14102025-07-17T11:31:19.914ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
14112025-07-17T11:31:19.914ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
14122025-07-17T11:31:19.917ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 153.432503ms iteration = 12 status = 0
14132025-07-17T11:31:19.931ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
14142025-07-17T11:31:19.931ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 168.177937ms iteration = 34 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
14152025-07-17T11:31:19.932ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
14162025-07-17T11:31:19.932ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
14172025-07-17T11:31:19.932ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
14182025-07-17T11:31:19.933ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
14192025-07-17T11:31:19.938ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 0 sp_type = switch
14202025-07-17T11:31:19.939ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
14212025-07-17T11:31:19.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
14222025-07-17T11:31:19.941ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 0 sp_type = switch
14232025-07-17T11:31:19.941ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = switch
14242025-07-17T11:31:19.941ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = switch
14252025-07-17T11:31:19.942ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 178.771247ms iteration = 12 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
14262025-07-17T11:31:19.943ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 0 sp_type = switch start_ena = None
14272025-07-17T11:31:19.943ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
14282025-07-17T11:31:19.944ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "29f361a4-93d3-47ff-9d84-039767f2b7a8", "content-length": "81", "date": "Thu, 17 Jul 2025 11:31:19 GMT"} })
14292025-07-17T11:31:19.946ZDEBGnexus (BackgroundTasks): activation complete background_task = sp_ereport_ingester elapsed = 183.089673ms iteration = 12 status = {"errors":[],"sps":[]}
14302025-07-17T11:31:19.957ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:372
14312025-07-17T11:31:19.957ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 194.17297ms iteration = 12 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
14322025-07-17T11:31:19.999ZINFOnexus (BackgroundTasks): VPC route manager sled 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
14332025-07-17T11:31:19.999ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
14342025-07-17T11:31:19.999ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d7f677b-62bb-47a9-87a9-7f1ddc6343ec", "content-length": "372", "date": "Thu, 17 Jul 2025 11:31:19 GMT"} })
14352025-07-17T11:31:19.999ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14362025-07-17T11:31:20.000ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14372025-07-17T11:31:20.005ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
14382025-07-17T11:31:20.005ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
14392025-07-17T11:31:20.005ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 241.704127ms iteration = 12 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
14402025-07-17T11:31:20.024ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
14412025-07-17T11:31:20.025ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
14422025-07-17T11:31:20.027ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14432025-07-17T11:31:20.027ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14442025-07-17T11:31:20.038ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 274.555829ms iteration = 13 status = {"physical_disks_added":0}
14452025-07-17T11:31:20.041ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 151caa7c-6a05-4995-94b4-7946d428117d, rack_id: 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5, switch_location: "switch0", port_name: Name(Name("qsfp0")), port_settings_id: Some(2e30c7ae-26da-46e4-9127-aac194921f20) } rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0
14462025-07-17T11:31:20.041ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14472025-07-17T11:31:20.041ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14482025-07-17T11:31:20.042ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
14492025-07-17T11:31:20.043ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
14502025-07-17T11:31:20.045ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14512025-07-17T11:31:20.047ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
14522025-07-17T11:31:20.047ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
14532025-07-17T11:31:20.047ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
14542025-07-17T11:31:20.049ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14552025-07-17T11:31:20.051ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
14562025-07-17T11:31:20.051ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
14572025-07-17T11:31:20.051ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
14582025-07-17T11:31:20.053ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1832 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = Name(Name("qsfp0"))
14592025-07-17T11:31:20.053ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1859 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0 switch_port_id = Name(Name("qsfp0")) switch_port_settings_id = 2e30c7ae-26da-46e4-9127-aac194921f20
14602025-07-17T11:31:20.053ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
14612025-07-17T11:31:20.053ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
14622025-07-17T11:31:20.053ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
14632025-07-17T11:31:20.053ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
14642025-07-17T11:31:20.053ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
14652025-07-17T11:31:20.053ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2006 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
14662025-07-17T11:31:20.053ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
14672025-07-17T11:31:20.053ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2042 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
14682025-07-17T11:31:20.053ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
14692025-07-17T11:31:20.053ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14702025-07-17T11:31:20.054ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14712025-07-17T11:31:20.056ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0
14722025-07-17T11:31:20.056ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
14732025-07-17T11:31:20.058ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
14742025-07-17T11:31:20.058ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
14752025-07-17T11:31:20.064ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "5477f0fe-3231-423c-968a-74d8f157814e", "date": "Thu, 17 Jul 2025 11:31:19 GMT"} })
14762025-07-17T11:31:20.064ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14772025-07-17T11:31:20.065ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14782025-07-17T11:31:20.068ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14792025-07-17T11:31:20.068ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14802025-07-17T11:31:20.070ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
14812025-07-17T11:31:20.071ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
14822025-07-17T11:31:20.074ZINFOnexus (BackgroundTasks): VPC route manager sled 0ae78f24-48db-4cf7-91cb-8d5311f2a17f push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
14832025-07-17T11:31:20.075ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
14842025-07-17T11:31:20.084ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "3b96184d-6b4a-4da2-bec6-ef964610e8b6", "date": "Thu, 17 Jul 2025 11:31:19 GMT"} })
14852025-07-17T11:31:20.086ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 322.900457ms iteration = 12 status = {}
14862025-07-17T11:31:20.106ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
14872025-07-17T11:31:20.107ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "359b3e3d-d93f-4d1c-b8b9-db9e606f64f2", "content-length": "691", "date": "Thu, 17 Jul 2025 11:31:20 GMT"} })
14882025-07-17T11:31:20.107ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
14892025-07-17T11:31:20.107ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
14902025-07-17T11:31:20.107ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
14912025-07-17T11:31:20.108ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 344.950048ms iteration = 12 status = {}
14922025-07-17T11:31:20.179ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "bddbfb9c-0741-4ffa-98fb-dda56116fdd7", "date": "Thu, 17 Jul 2025 11:31:20 GMT"} })
14932025-07-17T11:31:20.179ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch = Switch0
14942025-07-17T11:31:20.179ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
14952025-07-17T11:31:20.186ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "3346344c-c23b-48f9-9703-f9b54e381848", "content-length": "682", "date": "Thu, 17 Jul 2025 11:31:19 GMT"} })
14962025-07-17T11:31:20.186ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14972025-07-17T11:31:20.186ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14982025-07-17T11:31:20.190ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14992025-07-17T11:31:20.190ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
15002025-07-17T11:31:20.194ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
15012025-07-17T11:31:20.194ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
15022025-07-17T11:31:20.196ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
15032025-07-17T11:31:20.196ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
15042025-07-17T11:31:20.203ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 440.466397ms iteration = 12 status = {}
15052025-07-17T11:31:21.692ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 342 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = 9eed087a-1982-4561-b148-d0eab70e7a55 response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
15062025-07-17T11:31:24.763ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 68 reason = Timeout
15072025-07-17T11:31:24.763ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
15082025-07-17T11:31:24.763ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
15092025-07-17T11:31:24.765ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15102025-07-17T11:31:24.768ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
15112025-07-17T11:31:24.768ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
15122025-07-17T11:31:24.768ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
15132025-07-17T11:31:24.769ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
15142025-07-17T11:31:24.769ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 6.578327ms iteration = 68 status = {"chicken_switches_updated":false}
15152025-07-17T11:31:29.764ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 35 reason = Timeout
15162025-07-17T11:31:29.764ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 35 reason = Timeout
15172025-07-17T11:31:29.764ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
15182025-07-17T11:31:29.764ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15192025-07-17T11:31:29.764ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 69 reason = Timeout
15202025-07-17T11:31:29.764ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
15212025-07-17T11:31:29.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15222025-07-17T11:31:29.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
15232025-07-17T11:31:29.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
15242025-07-17T11:31:29.766ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15252025-07-17T11:31:29.766ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
15262025-07-17T11:31:29.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
15272025-07-17T11:31:29.767ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15282025-07-17T11:31:29.768ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.581221ms iteration = 35 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
15292025-07-17T11:31:29.769ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
15302025-07-17T11:31:29.769ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15312025-07-17T11:31:29.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15322025-07-17T11:31:29.770ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
15332025-07-17T11:31:29.770ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
15342025-07-17T11:31:29.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
15352025-07-17T11:31:29.772ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15362025-07-17T11:31:29.772ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
15372025-07-17T11:31:29.772ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 8.402321ms iteration = 69 status = {"chicken_switches_updated":false}
15382025-07-17T11:31:29.772ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15392025-07-17T11:31:29.773ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15402025-07-17T11:31:29.776ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
15412025-07-17T11:31:29.776ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15422025-07-17T11:31:29.776ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15432025-07-17T11:31:29.786ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
15442025-07-17T11:31:29.786ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.016941ms iteration = 35 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
15452025-07-17T11:31:31.691ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 388 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = 5af7398b-1afd-4603-a220-5326a1cd2abd response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
15462025-07-17T11:31:34.763ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 70 reason = Timeout
15472025-07-17T11:31:34.764ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
15482025-07-17T11:31:34.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
15492025-07-17T11:31:34.766ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15502025-07-17T11:31:34.769ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
15512025-07-17T11:31:34.769ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
15522025-07-17T11:31:34.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
15532025-07-17T11:31:34.770ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
15542025-07-17T11:31:34.770ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 6.733479ms iteration = 70 status = {"chicken_switches_updated":false}
15552025-07-17T11:31:39.763ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 36 reason = Timeout
15562025-07-17T11:31:39.763ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 36 reason = Timeout
15572025-07-17T11:31:39.763ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
15582025-07-17T11:31:39.763ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15592025-07-17T11:31:39.763ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 71 reason = Timeout
15602025-07-17T11:31:39.763ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
15612025-07-17T11:31:39.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
15622025-07-17T11:31:39.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15632025-07-17T11:31:39.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
15642025-07-17T11:31:39.766ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15652025-07-17T11:31:39.766ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15662025-07-17T11:31:39.768ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
15672025-07-17T11:31:39.768ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
15682025-07-17T11:31:39.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
15692025-07-17T11:31:39.769ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
15702025-07-17T11:31:39.769ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15712025-07-17T11:31:39.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15722025-07-17T11:31:39.770ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
15732025-07-17T11:31:39.770ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 6.83326ms iteration = 71 status = {"chicken_switches_updated":false}
15742025-07-17T11:31:39.771ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15752025-07-17T11:31:39.771ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15762025-07-17T11:31:39.772ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15772025-07-17T11:31:39.775ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
15782025-07-17T11:31:39.775ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15792025-07-17T11:31:39.775ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15802025-07-17T11:31:39.790ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
15812025-07-17T11:31:39.790ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 26.767513ms iteration = 36 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
15822025-07-17T11:31:39.799ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
15832025-07-17T11:31:39.799ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
15842025-07-17T11:31:39.839ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 75.748068ms iteration = 36 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
15852025-07-17T11:31:41.691ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 329 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = af69f0b8-70b8-4641-a3d9-73482b292e1e response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
15862025-07-17T11:31:44.764ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 72 reason = Timeout
15872025-07-17T11:31:44.764ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
15882025-07-17T11:31:44.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
15892025-07-17T11:31:44.766ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15902025-07-17T11:31:44.769ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
15912025-07-17T11:31:44.769ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
15922025-07-17T11:31:44.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
15932025-07-17T11:31:44.770ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
15942025-07-17T11:31:44.770ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 6.347403ms iteration = 72 status = {"chicken_switches_updated":false}
15952025-07-17T11:31:49.675ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 7 reason = Timeout
15962025-07-17T11:31:49.675ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 7 reason = Timeout
15972025-07-17T11:31:49.675ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 7 reason = Timeout
15982025-07-17T11:31:49.675ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-07-17 11:21:49.675579619 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
15992025-07-17T11:31:49.675ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
16002025-07-17T11:31:49.675ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 9 reason = Timeout
16012025-07-17T11:31:49.675ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
16022025-07-17T11:31:49.675ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 7 reason = Timeout
16032025-07-17T11:31:49.675ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
16042025-07-17T11:31:49.675ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 7 reason = Timeout
16052025-07-17T11:31:49.675ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
16062025-07-17T11:31:49.675ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 13 reason = Timeout
16072025-07-17T11:31:49.675ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
16082025-07-17T11:31:49.675ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 7 reason = Timeout
16092025-07-17T11:31:49.675ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
16102025-07-17T11:31:49.675ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
16112025-07-17T11:31:49.675ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 9 reason = Timeout
16122025-07-17T11:31:49.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
16132025-07-17T11:31:49.676ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("3dd3bf3b-7760-4750-b1d3-52ee777335f0.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("79e5c77d-18b3-4951-8d11-cfa3eae2b3e3.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("79e5c77d-18b3-4951-8d11-cfa3eae2b3e3.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::8)) }], valid_until: Instant { tv_sec: 3703, tv_nsec: 428983370 } })
16142025-07-17T11:31:49.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
16152025-07-17T11:31:49.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
16162025-07-17T11:31:49.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
16172025-07-17T11:31:49.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
16182025-07-17T11:31:49.676ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("3144a75a-317f-4b68-93e2-a210ca1189d1.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("87408888-9df4-4e0c-bc8e-2f920d039100.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("ee4378c5-e5f2-41cf-8969-e5e03e1b9a41.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("ee4378c5-e5f2-41cf-8969-e5e03e1b9a41.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:3::1)) }], valid_until: Instant { tv_sec: 3703, tv_nsec: 429124202 } })
16192025-07-17T11:31:49.676ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
16202025-07-17T11:31:49.676ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.061274ms iteration = 7 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
16212025-07-17T11:31:49.676ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } dns_group = internal
16222025-07-17T11:31:49.676ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.299288ms iteration = 7 status = {"addresses":["[fd00:1122:3344:1::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:3::1]:5353"]}
16232025-07-17T11:31:49.678ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 8 reason = Timeout
16242025-07-17T11:31:49.678ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:85 reason = blueprint disabled target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a
16252025-07-17T11:31:49.678ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 24.19\u{b5}s iteration = 8 status = {"enabled":false,"target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a"}
16262025-07-17T11:31:49.678ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16272025-07-17T11:31:49.679ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
16282025-07-17T11:31:49.679ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
16292025-07-17T11:31:49.679ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
16302025-07-17T11:31:49.681ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
16312025-07-17T11:31:49.681ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
16322025-07-17T11:31:49.682ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
16332025-07-17T11:31:49.682ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
16342025-07-17T11:31:49.682ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-07-17 11:21:49.675579619 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
16352025-07-17T11:31:49.682ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 6.7652ms iteration = 7 status = {"expiration":"2025-07-17T11:21:49.675579619Z","pruned":[]}
16362025-07-17T11:31:49.684ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16372025-07-17T11:31:49.684ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16382025-07-17T11:31:49.684ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 7 reason = Timeout
16392025-07-17T11:31:49.684ZDEBGnexus (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 4 reason = Timeout
16402025-07-17T11:31:49.684ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:53
16412025-07-17T11:31:49.684ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
16422025-07-17T11:31:49.684ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
16432025-07-17T11:31:49.684ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 7 reason = Timeout
16442025-07-17T11:31:49.684ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
16452025-07-17T11:31:49.684ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_planner iteration = 9 reason = Timeout
16462025-07-17T11:31:49.684ZDEBGnexus (BackgroundTasks): blueprint planning disabled, doing nothing background_task = blueprint_planner
16472025-07-17T11:31:49.684ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_planner elapsed = 8.33\u{b5}s iteration = 9 status = "Disabled"
16482025-07-17T11:31:49.685ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
16492025-07-17T11:31:49.685ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
16502025-07-17T11:31:49.685ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
16512025-07-17T11:31:49.685ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
16522025-07-17T11:31:49.685ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
16532025-07-17T11:31:49.685ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
16542025-07-17T11:31:49.685ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
16552025-07-17T11:31:49.685ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
16562025-07-17T11:31:49.685ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
16572025-07-17T11:31:49.685ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
16582025-07-17T11:31:49.685ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
16592025-07-17T11:31:49.686ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 7 reason = Timeout
16602025-07-17T11:31:49.686ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
16612025-07-17T11:31:49.686ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
16622025-07-17T11:31:49.689ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
16632025-07-17T11:31:49.690ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16642025-07-17T11:31:49.690ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16652025-07-17T11:31:49.691ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
16662025-07-17T11:31:49.691ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
16672025-07-17T11:31:49.691ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
16682025-07-17T11:31:49.691ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
16692025-07-17T11:31:49.691ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
16702025-07-17T11:31:49.691ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
16712025-07-17T11:31:49.692ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 7 reason = Timeout
16722025-07-17T11:31:49.692ZDEBGnexus (BackgroundTasks): activating background_task = webhook_deliverator iteration = 7 reason = Timeout
16732025-07-17T11:31:49.692ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
16742025-07-17T11:31:49.692ZDEBGnexus (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} }
16752025-07-17T11:31:49.692ZDEBGnexus (BackgroundTasks): activating background_task = alert_dispatcher iteration = 7 reason = Timeout
16762025-07-17T11:31:49.692ZDEBGnexus (BackgroundTasks): roles background_task = alert_dispatcher roles = RoleSet { roles: {} }
16772025-07-17T11:31:49.692ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = alert_dispatcher resource = Database result = Ok(())
16782025-07-17T11:31:49.692ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database result = Ok(())
16792025-07-17T11:31:49.692ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
16802025-07-17T11:31:49.693ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
16812025-07-17T11:31:49.693ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
16822025-07-17T11:31:49.693ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
16832025-07-17T11:31:49.694ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
16842025-07-17T11:31:49.694ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
16852025-07-17T11:31:49.694ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
16862025-07-17T11:31:49.698ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
16872025-07-17T11:31:49.698ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
16882025-07-17T11:31:49.698ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
16892025-07-17T11:31:49.698ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
16902025-07-17T11:31:49.698ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
16912025-07-17T11:31:49.698ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16922025-07-17T11:31:49.701ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
16932025-07-17T11:31:49.701ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
16942025-07-17T11:31:49.701ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
16952025-07-17T11:31:49.701ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3703, tv_nsec: 454264864 } })
16962025-07-17T11:31:49.701ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
16972025-07-17T11:31:49.701ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1186
16982025-07-17T11:31:49.702ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
16992025-07-17T11:31:49.706ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17002025-07-17T11:31:49.708ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17012025-07-17T11:31:49.709ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
17022025-07-17T11:31:49.709ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
17032025-07-17T11:31:49.709ZINFOnexus (BackgroundTasks): all webhook delivery tasks completed background_task = webhook_deliverator file = nexus/src/app/background/tasks/webhook_deliverator.rs:204 num_receivers = 0
17042025-07-17T11:31:49.709ZDEBGnexus (BackgroundTasks): activation complete background_task = webhook_deliverator elapsed = 16.821862ms iteration = 7 status = {"by_rx":{},"error":null}
17052025-07-17T11:31:49.709ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
17062025-07-17T11:31:49.709ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17072025-07-17T11:31:49.710ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
17082025-07-17T11:31:49.710ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:170 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
17092025-07-17T11:31:49.710ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 23.974497ms iteration = 7 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
17102025-07-17T11:31:49.710ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 17.939356ms iteration = 7 status = {"errors":[],"found_port_ok":[]}
17112025-07-17T11:31:49.712ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
17122025-07-17T11:31:49.712ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
17132025-07-17T11:31:49.712ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
17142025-07-17T11:31:49.713ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
17152025-07-17T11:31:49.713ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
17162025-07-17T11:31:49.713ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
17172025-07-17T11:31:49.715ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17182025-07-17T11:31:49.716ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17192025-07-17T11:31:49.718ZDEBGnexus (BackgroundTasks): roles background_task = alert_dispatcher roles = RoleSet { roles: {} }
17202025-07-17T11:31:49.718ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
17212025-07-17T11:31:49.718ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 4a9e4c5e-728e-4850-bd58-14548c02bf6f dns_zone_name = oxide.test found_so_far = 0 version = 2
17222025-07-17T11:31:49.718ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
17232025-07-17T11:31:49.719ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = alert_dispatcher resource = Database result = Ok(())
17242025-07-17T11:31:49.719ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
17252025-07-17T11:31:49.721ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17262025-07-17T11:31:49.722ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
17272025-07-17T11:31:49.722ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
17282025-07-17T11:31:49.722ZDEBGnexus (BackgroundTasks): activation complete background_task = alert_dispatcher elapsed = 30.152458ms iteration = 7 status = {"dispatched":[],"errors":[],"glob_version":"161.0.0","globs_reprocessed":{},"no_receivers":[]}
17292025-07-17T11:31:49.722ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
17302025-07-17T11:31:49.722ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
17312025-07-17T11:31:49.722ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
17322025-07-17T11:31:49.722ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
17332025-07-17T11:31:49.726ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
17342025-07-17T11:31:49.726ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
17352025-07-17T11:31:49.726ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
17362025-07-17T11:31:49.729ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 4a9e4c5e-728e-4850-bd58-14548c02bf6f dns_zone_name = oxide.test found_so_far = 4 version = 2
17372025-07-17T11:31:49.729ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
17382025-07-17T11:31:49.729ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-07-17 11:26:35.847594 UTC dns_group = external generation = 2
17392025-07-17T11:31:49.729ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 53.782579ms iteration = 7 status = {"generation":2}
17402025-07-17T11:31:49.732ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 13 reason = Timeout
17412025-07-17T11:31:49.732ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 13 reason = Timeout
17422025-07-17T11:31:49.732ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
17432025-07-17T11:31:49.732ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
17442025-07-17T11:31:49.744ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3703, tv_nsec: 497358492 } })
17452025-07-17T11:31:49.744ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 12.320382ms iteration = 13 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
17462025-07-17T11:31:49.744ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1186
17472025-07-17T11:31:49.763ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 13 reason = Timeout
17482025-07-17T11:31:49.763ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
17492025-07-17T11:31:49.763ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 13 reason = Timeout
17502025-07-17T11:31:49.763ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
17512025-07-17T11:31:49.763ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 13 reason = Timeout
17522025-07-17T11:31:49.763ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 13 reason = Timeout
17532025-07-17T11:31:49.763ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
17542025-07-17T11:31:49.763ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
17552025-07-17T11:31:49.763ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 37 reason = Timeout
17562025-07-17T11:31:49.763ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 13 reason = Timeout
17572025-07-17T11:31:49.763ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 13 reason = Timeout
17582025-07-17T11:31:49.763ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
17592025-07-17T11:31:49.763ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
17602025-07-17T11:31:49.763ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 14 reason = Timeout
17612025-07-17T11:31:49.763ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
17622025-07-17T11:31:49.763ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
17632025-07-17T11:31:49.763ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 13 reason = Timeout
17642025-07-17T11:31:49.763ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
17652025-07-17T11:31:49.763ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
17662025-07-17T11:31:49.763ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 13 reason = Timeout
17672025-07-17T11:31:49.763ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
17682025-07-17T11:31:49.763ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
17692025-07-17T11:31:49.763ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 13 reason = Timeout
17702025-07-17T11:31:49.763ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
17712025-07-17T11:31:49.763ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 13 reason = Timeout
17722025-07-17T11:31:49.763ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
17732025-07-17T11:31:49.764ZDEBGnexus (BackgroundTasks): activating background_task = sp_ereport_ingester iteration = 13 reason = Timeout
17742025-07-17T11:31:49.764ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 13 reason = Timeout
17752025-07-17T11:31:49.764ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
17762025-07-17T11:31:49.764ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 13 reason = Timeout
17772025-07-17T11:31:49.764ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
17782025-07-17T11:31:49.764ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 13 reason = Timeout
17792025-07-17T11:31:49.764ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 37 reason = Timeout
17802025-07-17T11:31:49.764ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
17812025-07-17T11:31:49.764ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
17822025-07-17T11:31:49.764ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 73 reason = Timeout
17832025-07-17T11:31:49.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
17842025-07-17T11:31:49.764ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
17852025-07-17T11:31:49.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
17862025-07-17T11:31:49.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
17872025-07-17T11:31:49.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
17882025-07-17T11:31:49.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
17892025-07-17T11:31:49.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
17902025-07-17T11:31:49.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
17912025-07-17T11:31:49.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17922025-07-17T11:31:49.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
17932025-07-17T11:31:49.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
17942025-07-17T11:31:49.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
17952025-07-17T11:31:49.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
17962025-07-17T11:31:49.764ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12225, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3703, tv_nsec: 517468607 } })
17972025-07-17T11:31:49.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
17982025-07-17T11:31:49.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
17992025-07-17T11:31:49.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
18002025-07-17T11:31:49.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
18012025-07-17T11:31:49.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
18022025-07-17T11:31:49.766ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18032025-07-17T11:31:49.766ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 2.98561ms iteration = 13 status = {"errors":[],"garbage_collect_requested":[]}
18042025-07-17T11:31:49.767ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18052025-07-17T11:31:49.767ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18062025-07-17T11:31:49.769ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18072025-07-17T11:31:49.770ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18082025-07-17T11:31:49.771ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
18092025-07-17T11:31:49.771ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
18102025-07-17T11:31:49.771ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
18112025-07-17T11:31:49.772ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
18122025-07-17T11:31:49.772ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
18132025-07-17T11:31:49.772ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
18142025-07-17T11:31:49.772ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
18152025-07-17T11:31:49.772ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
18162025-07-17T11:31:49.773ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
18172025-07-17T11:31:49.773ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
18182025-07-17T11:31:49.773ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
18192025-07-17T11:31:49.773ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
18202025-07-17T11:31:49.773ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
18212025-07-17T11:31:49.774ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
18222025-07-17T11:31:49.774ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
18232025-07-17T11:31:49.774ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18242025-07-17T11:31:49.774ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
18252025-07-17T11:31:49.777ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
18262025-07-17T11:31:49.777ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
18272025-07-17T11:31:49.777ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
18282025-07-17T11:31:49.778ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18292025-07-17T11:31:49.779ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
18302025-07-17T11:31:49.779ZDEBGnexus (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 94.937712ms iteration = 4 status = {"nsuccess":5}
18312025-07-17T11:31:49.779ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
18322025-07-17T11:31:49.779ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18332025-07-17T11:31:49.779ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
18342025-07-17T11:31:49.780ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
18352025-07-17T11:31:49.781ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
18362025-07-17T11:31:49.781ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
18372025-07-17T11:31:49.781ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
18382025-07-17T11:31:49.783ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18392025-07-17T11:31:49.786ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18402025-07-17T11:31:49.787ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
18412025-07-17T11:31:49.787ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
18422025-07-17T11:31:49.787ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
18432025-07-17T11:31:49.788ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
18442025-07-17T11:31:49.789ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18452025-07-17T11:31:49.790ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
18462025-07-17T11:31:49.790ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
18472025-07-17T11:31:49.790ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
18482025-07-17T11:31:49.790ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
18492025-07-17T11:31:49.793ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:550 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
18502025-07-17T11:31:49.793ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 29.637941ms iteration = 13 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
18512025-07-17T11:31:49.803ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
18522025-07-17T11:31:49.803ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
18532025-07-17T11:31:49.803ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
18542025-07-17T11:31:49.803ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
18552025-07-17T11:31:49.804ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3703, tv_nsec: 557201970 } })
18562025-07-17T11:31:49.804ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1186 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
18572025-07-17T11:31:49.808ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18582025-07-17T11:31:49.811ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
18592025-07-17T11:31:49.811ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
18602025-07-17T11:31:49.812ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
18612025-07-17T11:31:49.816ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
18622025-07-17T11:31:49.816ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:138 found = 0 not_ready_to_be_deleted = 0
18632025-07-17T11:31:49.816ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 131.522024ms iteration = 7 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
18642025-07-17T11:31:49.818ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
18652025-07-17T11:31:49.818ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 915fbba0-947f-4bf0-9f9f-48a79df81f7b dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
18662025-07-17T11:31:49.818ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
18672025-07-17T11:31:49.818ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
18682025-07-17T11:31:49.819ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
18692025-07-17T11:31:49.819ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
18702025-07-17T11:31:49.821ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18712025-07-17T11:31:49.821ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18722025-07-17T11:31:49.827ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
18732025-07-17T11:31:49.827ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
18742025-07-17T11:31:49.827ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
18752025-07-17T11:31:49.829ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
18762025-07-17T11:31:49.829ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
18772025-07-17T11:31:49.830ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
18782025-07-17T11:31:49.836ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 915fbba0-947f-4bf0-9f9f-48a79df81f7b dns_zone_name = control-plane.oxide.internal found_so_far = 59 version = 1
18792025-07-17T11:31:49.836ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
18802025-07-17T11:31:49.836ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-07-17 11:26:34.922237 UTC dns_group = internal generation = 1
18812025-07-17T11:31:49.836ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 160.626168ms iteration = 7 status = {"generation":1}
18822025-07-17T11:31:49.861ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18832025-07-17T11:31:49.865ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
18842025-07-17T11:31:49.865ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
18852025-07-17T11:31:49.866ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
18862025-07-17T11:31:49.871ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
18872025-07-17T11:31:49.871ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
18882025-07-17T11:31:49.874ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
18892025-07-17T11:31:49.874ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 110.74951ms iteration = 73 status = {"chicken_switches_updated":false}
18902025-07-17T11:31:49.874ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
18912025-07-17T11:31:49.875ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
18922025-07-17T11:31:49.875ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18932025-07-17T11:31:49.879ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
18942025-07-17T11:31:49.879ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
18952025-07-17T11:31:49.879ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
18962025-07-17T11:31:49.886ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
18972025-07-17T11:31:49.886ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
18982025-07-17T11:31:49.886ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
18992025-07-17T11:31:49.887ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
19002025-07-17T11:31:49.891ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 127.739364ms iteration = 13 status = {"errors":[],"finish_invoked_ok":[]}
19012025-07-17T11:31:49.891ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
19022025-07-17T11:31:49.892ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
19032025-07-17T11:31:49.897ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
19042025-07-17T11:31:49.897ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
19052025-07-17T11:31:49.899ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 135.908662ms iteration = 13 status = {"errors":[],"requests_created_ok":[]}
19062025-07-17T11:31:49.900ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 136.074044ms iteration = 13 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
19072025-07-17T11:31:49.906ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
19082025-07-17T11:31:49.907ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
19092025-07-17T11:31:49.921ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
19102025-07-17T11:31:49.922ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
19112025-07-17T11:31:49.925ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
19122025-07-17T11:31:49.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
19132025-07-17T11:31:49.938ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 174.751364ms iteration = 14 status = {"physical_disks_added":0}
19142025-07-17T11:31:49.938ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
19152025-07-17T11:31:49.940ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19162025-07-17T11:31:49.944ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
19172025-07-17T11:31:49.944ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
19182025-07-17T11:31:49.945ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
19192025-07-17T11:31:49.948ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
19202025-07-17T11:31:49.948ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
19212025-07-17T11:31:49.949ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1194 zone_address = fd00:1122:3344:101::2
19222025-07-17T11:31:49.949ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
19232025-07-17T11:31:49.949ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "22f17764-a3a6-4a74-a2cd-ddda4273b3ae", "content-length": "26", "date": "Thu, 17 Jul 2025 11:31:49 GMT"} })
19242025-07-17T11:31:49.949ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1197 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
19252025-07-17T11:31:49.950ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-07-17T11:25:46.964149Z, time_modified: 2025-07-17T11:25:46.964149Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 1fa836d9-7854-4ada-8c89-c26d922646c6, db_silo: Silo { identity: SiloIdentity { id: 1fa836d9-7854-4ada-8c89-c26d922646c6, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-07-17T11:26:35.643175Z, time_modified: 2025-07-17T11:26:35.643175Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "d7c3455b70a66c86ec2a7e597ca46fe0ea0c49dde5c1363f198c20aa77d58442" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 1fa836d9-7854-4ada-8c89-c26d922646c6, db_silo: Silo { identity: SiloIdentity { id: 1fa836d9-7854-4ada-8c89-c26d922646c6, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-07-17T11:26:35.643175Z, time_modified: 2025-07-17T11:26:35.643175Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "d7c3455b70a66c86ec2a7e597ca46fe0ea0c49dde5c1363f198c20aa77d58442" }] }) }
19262025-07-17T11:31:49.950ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 275.004226ms iteration = 7 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"1fa836d9-7854-4ada-8c89-c26d922646c6","tls_certs":["d7c3455b70a66c86ec2a7e597ca46fe0ea0c49dde5c1363f198c20aa77d58442"]}},"default_endpoint":{"silo_id":"1fa836d9-7854-4ada-8c89-c26d922646c6","tls_certs":["d7c3455b70a66c86ec2a7e597ca46fe0ea0c49dde5c1363f198c20aa77d58442"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
19272025-07-17T11:31:49.951ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
19282025-07-17T11:31:49.952ZINFOnexus (BackgroundTasks): instance reincarnation completed successfully background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:113 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0
19292025-07-17T11:31:49.952ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 267.561007ms iteration = 7 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
19302025-07-17T11:31:49.956ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 192.150684ms iteration = 37 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
19312025-07-17T11:31:49.962ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
19322025-07-17T11:31:49.962ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
19332025-07-17T11:31:49.962ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 198.660479ms iteration = 13 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
19342025-07-17T11:31:49.967ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
19352025-07-17T11:31:49.967ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 204.145731ms iteration = 37 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
19362025-07-17T11:31:49.974ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:372
19372025-07-17T11:31:49.974ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 210.329582ms iteration = 13 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
19382025-07-17T11:31:49.984ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 220.107162ms iteration = 13 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
19392025-07-17T11:31:49.985ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 221.632222ms iteration = 13 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
19402025-07-17T11:31:50.010ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19412025-07-17T11:31:50.011ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1194 zone_address = fd00:1122:3344:101::2
19422025-07-17T11:31:50.011ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
19432025-07-17T11:31:50.011ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "023fa331-c924-4a21-b4b3-0b7c88c6a49e", "content-length": "26", "date": "Thu, 17 Jul 2025 11:31:49 GMT"} })
19442025-07-17T11:31:50.011ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1197 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
19452025-07-17T11:31:50.013ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
19462025-07-17T11:31:50.013ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
19472025-07-17T11:31:50.017ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 0ae78f24-48db-4cf7-91cb-8d5311f2a17f, lookup_type: ById(0ae78f24-48db-4cf7-91cb-8d5311f2a17f) } result = Ok(())
19482025-07-17T11:31:50.017ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
19492025-07-17T11:31:50.017ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
19502025-07-17T11:31:50.021ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 257.313013ms iteration = 13 status = 0
19512025-07-17T11:31:50.025ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
19522025-07-17T11:31:50.031ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
19532025-07-17T11:31:50.031ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
19542025-07-17T11:31:50.033ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19552025-07-17T11:31:50.036ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
19562025-07-17T11:31:50.036ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
19572025-07-17T11:31:50.036ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
19582025-07-17T11:31:50.037ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19592025-07-17T11:31:50.039ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
19602025-07-17T11:31:50.039ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
19612025-07-17T11:31:50.040ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
19622025-07-17T11:31:50.172ZINFOnexus (BackgroundTasks): VPC route manager sled 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
19632025-07-17T11:31:50.172ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
19642025-07-17T11:31:50.172ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd4827ac-6ee8-4271-95fc-a36fb8691a70", "content-length": "372", "date": "Thu, 17 Jul 2025 11:31:50 GMT"} })
19652025-07-17T11:31:50.172ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
19662025-07-17T11:31:50.173ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
19672025-07-17T11:31:50.190ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
19682025-07-17T11:31:50.190ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
19692025-07-17T11:31:50.191ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "7db42935-da33-4a6e-b3c8-2c92682ad583", "date": "Thu, 17 Jul 2025 11:31:50 GMT"} })
19702025-07-17T11:31:50.191ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
19712025-07-17T11:31:50.191ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
19722025-07-17T11:31:50.192ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/all-sp-ids
19732025-07-17T11:31:50.193ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/all-sp-ids", status: 200, headers: {"content-type": "application/json", "x-request-id": "59fab43c-df06-46be-9366-5b05c1e9531c", "content-length": "78", "date": "Thu, 17 Jul 2025 11:31:49 GMT"} })
19742025-07-17T11:31:50.193ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = switch
19752025-07-17T11:31:50.193ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = sled
19762025-07-17T11:31:50.193ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 1 sp_type = sled
19772025-07-17T11:31:50.193ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = sled
19782025-07-17T11:31:50.193ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 1 sp_type = sled
19792025-07-17T11:31:50.193ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = switch
19802025-07-17T11:31:50.194ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
19812025-07-17T11:31:50.194ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
19822025-07-17T11:31:50.195ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 0 sp_type = sled
19832025-07-17T11:31:50.196ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 0 sp_type = switch
19842025-07-17T11:31:50.196ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 1 sp_type = sled
19852025-07-17T11:31:50.198ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 0 sp_type = sled
19862025-07-17T11:31:50.198ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = sled
19872025-07-17T11:31:50.198ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = sled
19882025-07-17T11:31:50.198ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
19892025-07-17T11:31:50.198ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
19902025-07-17T11:31:50.199ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 0 sp_type = switch
19912025-07-17T11:31:50.199ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = switch
19922025-07-17T11:31:50.200ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = switch
19932025-07-17T11:31:50.200ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 1 sp_type = sled
19942025-07-17T11:31:50.200ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 0 sp_type = sled start_ena = None
19952025-07-17T11:31:50.200ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 1 sp_type = sled
19962025-07-17T11:31:50.200ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
19972025-07-17T11:31:50.200ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 1 sp_type = sled
19982025-07-17T11:31:50.200ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "d60e6265-9952-45e8-ba55-b0d7cc615d53", "content-length": "81", "date": "Thu, 17 Jul 2025 11:31:49 GMT"} })
19992025-07-17T11:31:50.202ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 1 sp_type = sled start_ena = None
20002025-07-17T11:31:50.202ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/1/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
20012025-07-17T11:31:50.202ZINFOnexus (BackgroundTasks): VPC route manager sled 0ae78f24-48db-4cf7-91cb-8d5311f2a17f push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
20022025-07-17T11:31:50.202ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
20032025-07-17T11:31:50.202ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/1/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "7414e95c-40a7-4f06-a6db-b39ee43df733", "content-length": "81", "date": "Thu, 17 Jul 2025 11:31:49 GMT"} })
20042025-07-17T11:31:50.203ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "037ad571-aed8-465f-91f0-a917b5b3c446", "date": "Thu, 17 Jul 2025 11:31:50 GMT"} })
20052025-07-17T11:31:50.204ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 440.861402ms iteration = 13 status = {}
20062025-07-17T11:31:50.217ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
20072025-07-17T11:31:50.217ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
20082025-07-17T11:31:50.219ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 543.819269ms iteration = 13 status = null
20092025-07-17T11:31:50.221ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
20102025-07-17T11:31:50.238ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 0 sp_type = switch start_ena = None
20112025-07-17T11:31:50.238ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
20122025-07-17T11:31:50.238ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "5151e7c6-e55a-463e-818c-2db6fdf49437", "content-length": "81", "date": "Thu, 17 Jul 2025 11:31:49 GMT"} })
20132025-07-17T11:31:50.240ZDEBGnexus (BackgroundTasks): activation complete background_task = sp_ereport_ingester elapsed = 476.32723ms iteration = 13 status = {"errors":[],"sps":[]}
20142025-07-17T11:31:50.271ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
20152025-07-17T11:31:50.271ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "0142fbaf-1939-4524-b8c5-399816e05352", "content-length": "691", "date": "Thu, 17 Jul 2025 11:31:50 GMT"} })
20162025-07-17T11:31:50.271ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
20172025-07-17T11:31:50.271ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
20182025-07-17T11:31:50.271ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
20192025-07-17T11:31:50.273ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 510.046505ms iteration = 13 status = {}
20202025-07-17T11:31:50.277ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
20212025-07-17T11:31:50.278ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "6d2f66e3-07e9-49e8-b50f-c089d60cde3e", "date": "Thu, 17 Jul 2025 11:31:50 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
20222025-07-17T11:31:50.278ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
20232025-07-17T11:31:50.280ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "5f706b35-fc3e-40f9-9761-e5d9c012c8be", "date": "Thu, 17 Jul 2025 11:31:50 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
20242025-07-17T11:31:50.280ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
20252025-07-17T11:31:50.282ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
20262025-07-17T11:31:50.282ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 606.913961ms iteration = 9 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
20272025-07-17T11:31:50.284ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1194 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 zone_address = fd00:1122:3344:101::2
20282025-07-17T11:31:50.284ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
20292025-07-17T11:31:50.284ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "93c721ff-fcaa-42a9-a223-f7bc1aeb1382", "content-length": "26", "date": "Thu, 17 Jul 2025 11:31:49 GMT"} })
20302025-07-17T11:31:50.284ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1197 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
20312025-07-17T11:31:50.286ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
20322025-07-17T11:31:50.316ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 584.285453ms iteration = 13 status = {}
20332025-07-17T11:31:50.333ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
20342025-07-17T11:31:50.334ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "ce340a30-d36f-469f-a5b2-e2e741e6ff2d", "date": "Thu, 17 Jul 2025 11:31:50 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
20352025-07-17T11:31:50.334ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
20362025-07-17T11:31:50.336ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "d2ff6743-2b22-4265-b025-56bf773f22b2", "date": "Thu, 17 Jul 2025 11:31:50 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
20372025-07-17T11:31:50.336ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
20382025-07-17T11:31:50.338ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "eef3672c-3475-4204-89ca-4fe1a7f11125", "date": "Thu, 17 Jul 2025 11:31:50 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
20392025-07-17T11:31:50.338ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
20402025-07-17T11:31:50.340ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
20412025-07-17T11:31:50.340ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 664.680863ms iteration = 9 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
20422025-07-17T11:31:50.407ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
20432025-07-17T11:31:50.407ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
20442025-07-17T11:31:50.409ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 151caa7c-6a05-4995-94b4-7946d428117d, rack_id: 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5, switch_location: "switch0", port_name: Name(Name("qsfp0")), port_settings_id: Some(2e30c7ae-26da-46e4-9127-aac194921f20) } rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0
20452025-07-17T11:31:50.409ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
20462025-07-17T11:31:50.409ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
20472025-07-17T11:31:50.420ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1832 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = Name(Name("qsfp0"))
20482025-07-17T11:31:50.420ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1859 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0 switch_port_id = Name(Name("qsfp0")) switch_port_settings_id = 2e30c7ae-26da-46e4-9127-aac194921f20
20492025-07-17T11:31:50.420ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
20502025-07-17T11:31:50.420ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
20512025-07-17T11:31:50.420ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
20522025-07-17T11:31:50.420ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
20532025-07-17T11:31:50.420ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
20542025-07-17T11:31:50.420ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2006 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
20552025-07-17T11:31:50.420ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
20562025-07-17T11:31:50.420ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2042 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
20572025-07-17T11:31:50.420ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
20582025-07-17T11:31:50.420ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
20592025-07-17T11:31:50.421ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
20602025-07-17T11:31:50.423ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0
20612025-07-17T11:31:50.423ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
20622025-07-17T11:31:50.447ZDEBGnexus (producer-registration-task): registering as oximeter producer with Nexus registration_count = 2
20632025-07-17T11:31:50.447ZDEBGnexus (producer-registration-task): will register with Nexus at [fd00:1122:3344:101::b]:12221
20642025-07-17T11:31:50.479ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::b]:12221/metrics/producers
20652025-07-17T11:31:50.479ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::b]:50825
20662025-07-17T11:31:50.479ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:50825 req_id = 761f7652-b9e6-48db-9ecc-422e6f333182 roles = RoleSet { roles: {} } uri = /metrics/producers
20672025-07-17T11:31:50.479ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:50825 req_id = 761f7652-b9e6-48db-9ecc-422e6f333182 resource = Database result = Ok(()) uri = /metrics/producers
20682025-07-17T11:31:50.525ZINFOnexus: assigned collector to new producer collector_id = 3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d file = nexus/src/app/oximeter.rs:76 producer_id = 5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
20692025-07-17T11:31:50.525ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 45739 local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:50825 req_id = 761f7652-b9e6-48db-9ecc-422e6f333182 response_code = 201 uri = /metrics/producers
20702025-07-17T11:31:50.525ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "761f7652-b9e6-48db-9ecc-422e6f333182", "content-length": "41", "date": "Thu, 17 Jul 2025 11:31:50 GMT"} })
20712025-07-17T11:31:50.526ZDEBGnexus (producer-registration-task): successfully registered with Nexus, pausing until time to renew lease lease_duration = 600s wait_period = 150s
20722025-07-17T11:31:50.556ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "2fae6223-d20c-4338-acf1-65423afa7a21", "date": "Thu, 17 Jul 2025 11:31:50 GMT"} })
20732025-07-17T11:31:50.556ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch = Switch0
20742025-07-17T11:31:50.556ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
20752025-07-17T11:31:50.563ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "1287b67f-a714-4625-b703-86434ebae7a4", "content-length": "682", "date": "Thu, 17 Jul 2025 11:31:50 GMT"} })
20762025-07-17T11:31:50.563ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
20772025-07-17T11:31:50.564ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
20782025-07-17T11:31:50.567ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
20792025-07-17T11:31:50.568ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
20802025-07-17T11:31:50.570ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
20812025-07-17T11:31:50.570ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
20822025-07-17T11:31:50.572ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
20832025-07-17T11:31:50.572ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
20842025-07-17T11:31:50.580ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 816.265361ms iteration = 13 status = {}
20852025-07-17T11:31:51.692ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 373 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = 116a1e14-af11-462c-8363-abdd709e16e3 response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
20862025-07-17T11:31:54.764ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 74 reason = Timeout
20872025-07-17T11:31:54.764ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
20882025-07-17T11:31:54.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
20892025-07-17T11:31:54.766ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20902025-07-17T11:31:54.768ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
20912025-07-17T11:31:54.768ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
20922025-07-17T11:31:54.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
20932025-07-17T11:31:54.770ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
20942025-07-17T11:31:54.770ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 5.854178ms iteration = 74 status = {"chicken_switches_updated":false}
20952025-07-17T11:31:59.764ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 38 reason = Timeout
20962025-07-17T11:31:59.764ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
20972025-07-17T11:31:59.764ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 38 reason = Timeout
20982025-07-17T11:31:59.764ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 75 reason = Timeout
20992025-07-17T11:31:59.764ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21002025-07-17T11:31:59.764ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
21012025-07-17T11:31:59.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
21022025-07-17T11:31:59.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
21032025-07-17T11:31:59.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21042025-07-17T11:31:59.766ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21052025-07-17T11:31:59.767ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21062025-07-17T11:31:59.770ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
21072025-07-17T11:31:59.770ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
21082025-07-17T11:31:59.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
21092025-07-17T11:31:59.770ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
21102025-07-17T11:31:59.770ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21112025-07-17T11:31:59.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21122025-07-17T11:31:59.771ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
21132025-07-17T11:31:59.771ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 7.712072ms iteration = 75 status = {"chicken_switches_updated":false}
21142025-07-17T11:31:59.772ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21152025-07-17T11:31:59.772ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21162025-07-17T11:31:59.774ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21172025-07-17T11:31:59.777ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
21182025-07-17T11:31:59.777ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21192025-07-17T11:31:59.777ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21202025-07-17T11:31:59.788ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
21212025-07-17T11:31:59.788ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 24.670715ms iteration = 38 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
21222025-07-17T11:31:59.803ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
21232025-07-17T11:31:59.804ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
21242025-07-17T11:31:59.805ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 41.482867ms iteration = 38 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
21252025-07-17T11:32:01.691ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 345 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = b75c217d-8723-40a1-baff-5b9f16df4262 response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
21262025-07-17T11:32:04.764ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 76 reason = Timeout
21272025-07-17T11:32:04.764ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
21282025-07-17T11:32:04.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
21292025-07-17T11:32:04.766ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21302025-07-17T11:32:04.768ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
21312025-07-17T11:32:04.768ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
21322025-07-17T11:32:04.768ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
21332025-07-17T11:32:04.769ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
21342025-07-17T11:32:04.769ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 5.942799ms iteration = 76 status = {"chicken_switches_updated":false}
21352025-07-17T11:32:09.764ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 39 reason = Timeout
21362025-07-17T11:32:09.764ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 39 reason = Timeout
21372025-07-17T11:32:09.764ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
21382025-07-17T11:32:09.764ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21392025-07-17T11:32:09.764ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 77 reason = Timeout
21402025-07-17T11:32:09.764ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
21412025-07-17T11:32:09.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
21422025-07-17T11:32:09.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21432025-07-17T11:32:09.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
21442025-07-17T11:32:09.765ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
21452025-07-17T11:32:09.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
21462025-07-17T11:32:09.766ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21472025-07-17T11:32:09.766ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21482025-07-17T11:32:09.767ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.212692ms iteration = 39 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
21492025-07-17T11:32:09.769ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
21502025-07-17T11:32:09.770ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21512025-07-17T11:32:09.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21522025-07-17T11:32:09.770ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
21532025-07-17T11:32:09.770ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
21542025-07-17T11:32:09.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
21552025-07-17T11:32:09.771ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21562025-07-17T11:32:09.772ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
21572025-07-17T11:32:09.772ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 7.891554ms iteration = 77 status = {"chicken_switches_updated":false}
21582025-07-17T11:32:09.772ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21592025-07-17T11:32:09.776ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21602025-07-17T11:32:09.779ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
21612025-07-17T11:32:09.779ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21622025-07-17T11:32:09.779ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21632025-07-17T11:32:09.789ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
21642025-07-17T11:32:09.789ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 25.127791ms iteration = 39 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
21652025-07-17T11:32:11.692ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 324 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = 4ad0447b-bfb6-450e-bef3-a94114d8ec70 response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
21662025-07-17T11:32:14.763ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 78 reason = Timeout
21672025-07-17T11:32:14.763ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
21682025-07-17T11:32:14.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
21692025-07-17T11:32:14.765ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21702025-07-17T11:32:14.768ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
21712025-07-17T11:32:14.768ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
21722025-07-17T11:32:14.768ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
21732025-07-17T11:32:14.769ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
21742025-07-17T11:32:14.769ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 6.253802ms iteration = 78 status = {"chicken_switches_updated":false}
21752025-07-17T11:32:19.676ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 14 reason = Timeout
21762025-07-17T11:32:19.676ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
21772025-07-17T11:32:19.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
21782025-07-17T11:32:19.679ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3733, tv_nsec: 431912296 } })
21792025-07-17T11:32:19.679ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1186
21802025-07-17T11:32:19.709ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1194 zone_address = fd00:1122:3344:101::2
21812025-07-17T11:32:19.709ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
21822025-07-17T11:32:19.710ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5e23c04-244d-4143-8f0d-eeb70e88a4e3", "content-length": "26", "date": "Thu, 17 Jul 2025 11:32:19 GMT"} })
21832025-07-17T11:32:19.710ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1197 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
21842025-07-17T11:32:19.710ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
21852025-07-17T11:32:19.732ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 14 reason = Timeout
21862025-07-17T11:32:19.732ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 14 reason = Timeout
21872025-07-17T11:32:19.732ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
21882025-07-17T11:32:19.733ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
21892025-07-17T11:32:19.736ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.194212ms iteration = 14 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
21902025-07-17T11:32:19.736ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3733, tv_nsec: 489023778 } })
21912025-07-17T11:32:19.736ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1186
21922025-07-17T11:32:19.743ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
21932025-07-17T11:32:19.744ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
21942025-07-17T11:32:19.746ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 70.356548ms iteration = 14 status = null
21952025-07-17T11:32:19.764ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 14 reason = Timeout
21962025-07-17T11:32:19.764ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 14 reason = Timeout
21972025-07-17T11:32:19.764ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
21982025-07-17T11:32:19.764ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
21992025-07-17T11:32:19.764ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 14 reason = Timeout
22002025-07-17T11:32:19.764ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
22012025-07-17T11:32:19.764ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 14 reason = Timeout
22022025-07-17T11:32:19.764ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
22032025-07-17T11:32:19.764ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 14 reason = Timeout
22042025-07-17T11:32:19.764ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
22052025-07-17T11:32:19.764ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
22062025-07-17T11:32:19.764ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 14 reason = Timeout
22072025-07-17T11:32:19.764ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
22082025-07-17T11:32:19.764ZDEBGnexus (BackgroundTasks): activating background_task = sp_ereport_ingester iteration = 14 reason = Timeout
22092025-07-17T11:32:19.764ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 14 reason = Timeout
22102025-07-17T11:32:19.764ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
22112025-07-17T11:32:19.764ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
22122025-07-17T11:32:19.764ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 15 reason = Timeout
22132025-07-17T11:32:19.764ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
22142025-07-17T11:32:19.764ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 14 reason = Timeout
22152025-07-17T11:32:19.764ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 40 reason = Timeout
22162025-07-17T11:32:19.764ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
22172025-07-17T11:32:19.764ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
22182025-07-17T11:32:19.764ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 40 reason = Timeout
22192025-07-17T11:32:19.764ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
22202025-07-17T11:32:19.764ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 79 reason = Timeout
22212025-07-17T11:32:19.764ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 14 reason = Timeout
22222025-07-17T11:32:19.764ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
22232025-07-17T11:32:19.764ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
22242025-07-17T11:32:19.764ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 14 reason = Timeout
22252025-07-17T11:32:19.764ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
22262025-07-17T11:32:19.764ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 14 reason = Timeout
22272025-07-17T11:32:19.764ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
22282025-07-17T11:32:19.764ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 14 reason = Timeout
22292025-07-17T11:32:19.764ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
22302025-07-17T11:32:19.764ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 14 reason = Timeout
22312025-07-17T11:32:19.764ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
22322025-07-17T11:32:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
22332025-07-17T11:32:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
22342025-07-17T11:32:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
22352025-07-17T11:32:19.764ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12225, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3733, tv_nsec: 517344942 } })
22362025-07-17T11:32:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
22372025-07-17T11:32:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
22382025-07-17T11:32:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
22392025-07-17T11:32:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
22402025-07-17T11:32:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
22412025-07-17T11:32:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
22422025-07-17T11:32:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
22432025-07-17T11:32:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
22442025-07-17T11:32:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
22452025-07-17T11:32:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
22462025-07-17T11:32:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
22472025-07-17T11:32:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
22482025-07-17T11:32:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
22492025-07-17T11:32:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
22502025-07-17T11:32:19.767ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22512025-07-17T11:32:19.767ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22522025-07-17T11:32:19.768ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 4.133615ms iteration = 14 status = {"errors":[],"garbage_collect_requested":[]}
22532025-07-17T11:32:19.768ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
22542025-07-17T11:32:19.768ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
22552025-07-17T11:32:19.768ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
22562025-07-17T11:32:19.769ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
22572025-07-17T11:32:19.769ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22582025-07-17T11:32:19.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
22592025-07-17T11:32:19.769ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22602025-07-17T11:32:19.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
22612025-07-17T11:32:19.769ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
22622025-07-17T11:32:19.769ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
22632025-07-17T11:32:19.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
22642025-07-17T11:32:19.769ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22652025-07-17T11:32:19.770ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
22662025-07-17T11:32:19.770ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22672025-07-17T11:32:19.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
22682025-07-17T11:32:19.771ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
22692025-07-17T11:32:19.771ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
22702025-07-17T11:32:19.771ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22712025-07-17T11:32:19.771ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
22722025-07-17T11:32:19.771ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22732025-07-17T11:32:19.772ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 7.833513ms iteration = 14 status = {"errors":[],"finish_invoked_ok":[]}
22742025-07-17T11:32:19.772ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
22752025-07-17T11:32:19.772ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
22762025-07-17T11:32:19.772ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
22772025-07-17T11:32:19.772ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
22782025-07-17T11:32:19.772ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
22792025-07-17T11:32:19.773ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1194 zone_address = fd00:1122:3344:101::2
22802025-07-17T11:32:19.773ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
22812025-07-17T11:32:19.773ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2b93d3b-300e-4be7-a0ac-52de42034ac3", "content-length": "26", "date": "Thu, 17 Jul 2025 11:32:19 GMT"} })
22822025-07-17T11:32:19.773ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1197 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
22832025-07-17T11:32:19.774ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
22842025-07-17T11:32:19.774ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
22852025-07-17T11:32:19.774ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
22862025-07-17T11:32:19.774ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
22872025-07-17T11:32:19.774ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
22882025-07-17T11:32:19.775ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
22892025-07-17T11:32:19.775ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 11.500142ms iteration = 40 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
22902025-07-17T11:32:19.776ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
22912025-07-17T11:32:19.776ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
22922025-07-17T11:32:19.776ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
22932025-07-17T11:32:19.776ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
22942025-07-17T11:32:19.776ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
22952025-07-17T11:32:19.776ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
22962025-07-17T11:32:19.776ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
22972025-07-17T11:32:19.776ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
22982025-07-17T11:32:19.776ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
22992025-07-17T11:32:19.776ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
23002025-07-17T11:32:19.776ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23012025-07-17T11:32:19.777ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
23022025-07-17T11:32:19.777ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
23032025-07-17T11:32:19.778ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
23042025-07-17T11:32:19.778ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
23052025-07-17T11:32:19.779ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
23062025-07-17T11:32:19.779ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
23072025-07-17T11:32:19.779ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 15.0965ms iteration = 79 status = {"chicken_switches_updated":false}
23082025-07-17T11:32:19.779ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
23092025-07-17T11:32:19.779ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
23102025-07-17T11:32:19.779ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
23112025-07-17T11:32:19.780ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
23122025-07-17T11:32:19.780ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
23132025-07-17T11:32:19.780ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
23142025-07-17T11:32:19.781ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23152025-07-17T11:32:19.781ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3733, tv_nsec: 534025942 } })
23162025-07-17T11:32:19.781ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1186 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
23172025-07-17T11:32:19.783ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
23182025-07-17T11:32:19.783ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
23192025-07-17T11:32:19.784ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
23202025-07-17T11:32:19.784ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23212025-07-17T11:32:19.786ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23222025-07-17T11:32:19.786ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:550 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
23232025-07-17T11:32:19.786ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 20.788925ms iteration = 14 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
23242025-07-17T11:32:19.788ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
23252025-07-17T11:32:19.788ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
23262025-07-17T11:32:19.788ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
23272025-07-17T11:32:19.791ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
23282025-07-17T11:32:19.791ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
23292025-07-17T11:32:19.792ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
23302025-07-17T11:32:19.809ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
23312025-07-17T11:32:19.809ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 45.228917ms iteration = 40 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
23322025-07-17T11:32:19.836ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
23332025-07-17T11:32:19.836ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
23342025-07-17T11:32:19.836ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
23352025-07-17T11:32:19.836ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
23362025-07-17T11:32:19.838ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
23372025-07-17T11:32:19.838ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
23382025-07-17T11:32:19.840ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
23392025-07-17T11:32:19.840ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
23402025-07-17T11:32:19.840ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 76.548549ms iteration = 14 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
23412025-07-17T11:32:19.841ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/all-sp-ids
23422025-07-17T11:32:19.842ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/all-sp-ids", status: 200, headers: {"content-type": "application/json", "x-request-id": "285e4bce-820f-4646-bb9e-58f3ef919e7d", "content-length": "78", "date": "Thu, 17 Jul 2025 11:32:19 GMT"} })
23432025-07-17T11:32:19.842ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = switch
23442025-07-17T11:32:19.842ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 1 sp_type = sled
23452025-07-17T11:32:19.842ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = sled
23462025-07-17T11:32:19.842ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 1 sp_type = sled
23472025-07-17T11:32:19.842ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = switch
23482025-07-17T11:32:19.842ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = sled
23492025-07-17T11:32:19.844ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 80.044525ms iteration = 14 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
23502025-07-17T11:32:19.845ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
23512025-07-17T11:32:19.845ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 1 sp_type = sled
23522025-07-17T11:32:19.845ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 0 sp_type = switch
23532025-07-17T11:32:19.845ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
23542025-07-17T11:32:19.846ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 0 sp_type = sled
23552025-07-17T11:32:19.848ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 84.317322ms iteration = 14 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
23562025-07-17T11:32:19.849ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 1 sp_type = sled
23572025-07-17T11:32:19.849ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 1 sp_type = sled
23582025-07-17T11:32:19.849ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
23592025-07-17T11:32:19.849ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 0 sp_type = switch
23602025-07-17T11:32:19.849ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = switch
23612025-07-17T11:32:19.850ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 1 sp_type = sled
23622025-07-17T11:32:19.850ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = switch
23632025-07-17T11:32:19.850ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
23642025-07-17T11:32:19.850ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 0 sp_type = sled
23652025-07-17T11:32:19.850ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = sled
23662025-07-17T11:32:19.850ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = sled
23672025-07-17T11:32:19.852ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 1 sp_type = sled start_ena = None
23682025-07-17T11:32:19.852ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/1/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
23692025-07-17T11:32:19.852ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/1/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f3e32ef-7ce8-4656-9a79-503efbe16d2d", "content-length": "81", "date": "Thu, 17 Jul 2025 11:32:19 GMT"} })
23702025-07-17T11:32:19.855ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:372
23712025-07-17T11:32:19.855ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 91.134932ms iteration = 14 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
23722025-07-17T11:32:19.855ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
23732025-07-17T11:32:19.856ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
23742025-07-17T11:32:19.859ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
23752025-07-17T11:32:19.860ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
23762025-07-17T11:32:19.863ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23772025-07-17T11:32:19.866ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 133.807854ms iteration = 14 status = {}
23782025-07-17T11:32:19.867ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 103.080159ms iteration = 14 status = {"errors":[],"requests_created_ok":[]}
23792025-07-17T11:32:19.868ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 0ae78f24-48db-4cf7-91cb-8d5311f2a17f, lookup_type: ById(0ae78f24-48db-4cf7-91cb-8d5311f2a17f) } result = Ok(())
23802025-07-17T11:32:19.868ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
23812025-07-17T11:32:19.868ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
23822025-07-17T11:32:19.873ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 109.081988ms iteration = 14 status = 0
23832025-07-17T11:32:19.873ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
23842025-07-17T11:32:19.874ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
23852025-07-17T11:32:19.876ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23862025-07-17T11:32:19.878ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
23872025-07-17T11:32:19.878ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
23882025-07-17T11:32:19.878ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
23892025-07-17T11:32:19.883ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23902025-07-17T11:32:19.884ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1194 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 zone_address = fd00:1122:3344:101::2
23912025-07-17T11:32:19.884ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
23922025-07-17T11:32:19.885ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "21663a9b-bb8b-4535-8e14-993d9d8fff17", "content-length": "26", "date": "Thu, 17 Jul 2025 11:32:19 GMT"} })
23932025-07-17T11:32:19.885ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1197 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
23942025-07-17T11:32:19.885ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
23952025-07-17T11:32:19.885ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
23962025-07-17T11:32:19.886ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
23972025-07-17T11:32:19.887ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
23982025-07-17T11:32:19.905ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 141.09222ms iteration = 14 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
23992025-07-17T11:32:19.915ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 0 sp_type = switch start_ena = None
24002025-07-17T11:32:19.915ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
24012025-07-17T11:32:19.915ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "afe8319b-0261-4a0f-ac14-4509444741f7", "content-length": "81", "date": "Thu, 17 Jul 2025 11:32:19 GMT"} })
24022025-07-17T11:32:19.921ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 0 sp_type = sled start_ena = None
24032025-07-17T11:32:19.921ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
24042025-07-17T11:32:19.921ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "880c753d-115f-41a0-b416-564db5d47aee", "content-length": "81", "date": "Thu, 17 Jul 2025 11:32:19 GMT"} })
24052025-07-17T11:32:19.923ZDEBGnexus (BackgroundTasks): activation complete background_task = sp_ereport_ingester elapsed = 159.580974ms iteration = 14 status = {"errors":[],"sps":[]}
24062025-07-17T11:32:19.951ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
24072025-07-17T11:32:19.951ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb271f3b-b869-44f2-a1ba-49674200e5e4", "content-length": "691", "date": "Thu, 17 Jul 2025 11:32:19 GMT"} })
24082025-07-17T11:32:19.951ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
24092025-07-17T11:32:19.951ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
24102025-07-17T11:32:19.951ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
24112025-07-17T11:32:19.954ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 190.662633ms iteration = 14 status = {}
24122025-07-17T11:32:19.968ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 204.249653ms iteration = 15 status = {"physical_disks_added":0}
24132025-07-17T11:32:20.018ZINFOnexus (BackgroundTasks): VPC route manager sled 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
24142025-07-17T11:32:20.018ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
24152025-07-17T11:32:20.019ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ff88e1c-97d3-42f0-b299-6b622100dd19", "content-length": "372", "date": "Thu, 17 Jul 2025 11:32:19 GMT"} })
24162025-07-17T11:32:20.019ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
24172025-07-17T11:32:20.019ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
24182025-07-17T11:32:20.034ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
24192025-07-17T11:32:20.034ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
24202025-07-17T11:32:20.035ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "ff5a6b70-80e7-40c5-a718-148beaef8072", "date": "Thu, 17 Jul 2025 11:32:19 GMT"} })
24212025-07-17T11:32:20.035ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
24222025-07-17T11:32:20.035ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
24232025-07-17T11:32:20.037ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
24242025-07-17T11:32:20.038ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
24252025-07-17T11:32:20.041ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24262025-07-17T11:32:20.041ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
24272025-07-17T11:32:20.041ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
24282025-07-17T11:32:20.041ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24292025-07-17T11:32:20.043ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 151caa7c-6a05-4995-94b4-7946d428117d, rack_id: 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5, switch_location: "switch0", port_name: Name(Name("qsfp0")), port_settings_id: Some(2e30c7ae-26da-46e4-9127-aac194921f20) } rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0
24302025-07-17T11:32:20.043ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24312025-07-17T11:32:20.044ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24322025-07-17T11:32:20.044ZINFOnexus (BackgroundTasks): VPC route manager sled 0ae78f24-48db-4cf7-91cb-8d5311f2a17f push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
24332025-07-17T11:32:20.044ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
24342025-07-17T11:32:20.045ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "d0c6c161-1195-4f2d-bda5-a5ca6f055e95", "date": "Thu, 17 Jul 2025 11:32:19 GMT"} })
24352025-07-17T11:32:20.046ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 282.600345ms iteration = 14 status = {}
24362025-07-17T11:32:20.056ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1832 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = Name(Name("qsfp0"))
24372025-07-17T11:32:20.056ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1859 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0 switch_port_id = Name(Name("qsfp0")) switch_port_settings_id = 2e30c7ae-26da-46e4-9127-aac194921f20
24382025-07-17T11:32:20.057ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
24392025-07-17T11:32:20.057ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
24402025-07-17T11:32:20.057ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
24412025-07-17T11:32:20.057ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
24422025-07-17T11:32:20.057ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
24432025-07-17T11:32:20.057ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2006 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
24442025-07-17T11:32:20.057ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
24452025-07-17T11:32:20.057ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2042 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
24462025-07-17T11:32:20.057ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
24472025-07-17T11:32:20.057ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24482025-07-17T11:32:20.058ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24492025-07-17T11:32:20.060ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0
24502025-07-17T11:32:20.060ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
24512025-07-17T11:32:20.176ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "4f79eb1f-a964-45d3-9129-b124a7ce8135", "date": "Thu, 17 Jul 2025 11:32:20 GMT"} })
24522025-07-17T11:32:20.176ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch = Switch0
24532025-07-17T11:32:20.176ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
24542025-07-17T11:32:20.183ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "ebb11e4c-9b96-4bec-9df7-af4893d01b04", "content-length": "682", "date": "Thu, 17 Jul 2025 11:32:20 GMT"} })
24552025-07-17T11:32:20.184ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24562025-07-17T11:32:20.184ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24572025-07-17T11:32:20.188ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24582025-07-17T11:32:20.188ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24592025-07-17T11:32:20.191ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24602025-07-17T11:32:20.192ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24612025-07-17T11:32:20.194ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24622025-07-17T11:32:20.194ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24632025-07-17T11:32:20.201ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 437.129783ms iteration = 14 status = {}
24642025-07-17T11:32:21.691ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 351 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = a675138b-dea8-4395-878b-58ff2f1da944 response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
24652025-07-17T11:32:24.763ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 80 reason = Timeout
24662025-07-17T11:32:24.763ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
24672025-07-17T11:32:24.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
24682025-07-17T11:32:24.765ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24692025-07-17T11:32:24.768ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
24702025-07-17T11:32:24.768ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
24712025-07-17T11:32:24.768ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
24722025-07-17T11:32:24.770ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
24732025-07-17T11:32:24.770ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 6.575997ms iteration = 80 status = {"chicken_switches_updated":false}
24742025-07-17T11:32:29.763ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 41 reason = Timeout
24752025-07-17T11:32:29.763ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
24762025-07-17T11:32:29.763ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 41 reason = Timeout
24772025-07-17T11:32:29.763ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 81 reason = Timeout
24782025-07-17T11:32:29.763ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24792025-07-17T11:32:29.763ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
24802025-07-17T11:32:29.763ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
24812025-07-17T11:32:29.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24822025-07-17T11:32:29.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
24832025-07-17T11:32:29.765ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
24842025-07-17T11:32:29.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
24852025-07-17T11:32:29.765ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24862025-07-17T11:32:29.766ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24872025-07-17T11:32:29.767ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.73774ms iteration = 41 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
24882025-07-17T11:32:29.769ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
24892025-07-17T11:32:29.769ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24902025-07-17T11:32:29.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24912025-07-17T11:32:29.769ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
24922025-07-17T11:32:29.769ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
24932025-07-17T11:32:29.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
24942025-07-17T11:32:29.771ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24952025-07-17T11:32:29.771ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24962025-07-17T11:32:29.771ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
24972025-07-17T11:32:29.771ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 7.925254ms iteration = 81 status = {"chicken_switches_updated":false}
24982025-07-17T11:32:29.772ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24992025-07-17T11:32:29.775ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
25002025-07-17T11:32:29.775ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25012025-07-17T11:32:29.775ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25022025-07-17T11:32:29.784ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
25032025-07-17T11:32:29.784ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.424779ms iteration = 41 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
25042025-07-17T11:32:31.691ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 361 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = fb2e6609-127d-484f-b266-845ccfac4389 response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
25052025-07-17T11:32:34.763ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 82 reason = Timeout
25062025-07-17T11:32:34.763ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
25072025-07-17T11:32:34.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
25082025-07-17T11:32:34.765ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25092025-07-17T11:32:34.768ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
25102025-07-17T11:32:34.768ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
25112025-07-17T11:32:34.768ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
25122025-07-17T11:32:34.770ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
25132025-07-17T11:32:34.770ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 6.534846ms iteration = 82 status = {"chicken_switches_updated":false}
25142025-07-17T11:32:39.764ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 42 reason = Timeout
25152025-07-17T11:32:39.764ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
25162025-07-17T11:32:39.764ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 42 reason = Timeout
25172025-07-17T11:32:39.764ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 83 reason = Timeout
25182025-07-17T11:32:39.764ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25192025-07-17T11:32:39.764ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
25202025-07-17T11:32:39.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25212025-07-17T11:32:39.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
25222025-07-17T11:32:39.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
25232025-07-17T11:32:39.767ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25242025-07-17T11:32:39.767ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
25252025-07-17T11:32:39.767ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25262025-07-17T11:32:39.767ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
25272025-07-17T11:32:39.769ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.815174ms iteration = 42 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
25282025-07-17T11:32:39.770ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
25292025-07-17T11:32:39.770ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25302025-07-17T11:32:39.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25312025-07-17T11:32:39.770ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
25322025-07-17T11:32:39.770ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
25332025-07-17T11:32:39.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
25342025-07-17T11:32:39.772ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25352025-07-17T11:32:39.772ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25362025-07-17T11:32:39.772ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
25372025-07-17T11:32:39.772ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 7.982605ms iteration = 83 status = {"chicken_switches_updated":false}
25382025-07-17T11:32:39.773ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25392025-07-17T11:32:39.776ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
25402025-07-17T11:32:39.776ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25412025-07-17T11:32:39.776ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25422025-07-17T11:32:39.786ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
25432025-07-17T11:32:39.786ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.938049ms iteration = 42 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
25442025-07-17T11:32:41.692ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 323 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = e8f7141c-97ea-440e-b9b0-e62aa825491f response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
25452025-07-17T11:32:44.764ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 84 reason = Timeout
25462025-07-17T11:32:44.764ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
25472025-07-17T11:32:44.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
25482025-07-17T11:32:44.766ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25492025-07-17T11:32:44.769ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
25502025-07-17T11:32:44.769ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
25512025-07-17T11:32:44.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
25522025-07-17T11:32:44.772ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
25532025-07-17T11:32:44.772ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 7.728012ms iteration = 84 status = {"chicken_switches_updated":false}
25542025-07-17T11:32:46.683ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::d]:35131
25552025-07-17T11:32:46.683ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:35131 req_id = 91c38835-21f7-4ae1-8209-781760b0a150 roles = RoleSet { roles: {} } uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?sort_by=id_ascending
25562025-07-17T11:32:46.684ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:35131 req_id = 91c38835-21f7-4ae1-8209-781760b0a150 resource = Database result = Ok(()) uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?sort_by=id_ascending
25572025-07-17T11:32:46.687ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 3189 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:35131 req_id = 91c38835-21f7-4ae1-8209-781760b0a150 response_code = 200 uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?sort_by=id_ascending
25582025-07-17T11:32:46.687ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:35131 req_id = cd336562-3972-4eb8-a55f-7d7c46e756da roles = RoleSet { roles: {} } uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmMzlkM2U3MC02NmU0LTQ2OGEtYTQ0ZC03NDZkMjJiZWEzYTAifX0%3D
25592025-07-17T11:32:46.687ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:35131 req_id = cd336562-3972-4eb8-a55f-7d7c46e756da resource = Database result = Ok(()) uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmMzlkM2U3MC02NmU0LTQ2OGEtYTQ0ZC03NDZkMjJiZWEzYTAifX0%3D
25602025-07-17T11:32:46.689ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 1864 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:35131 req_id = cd336562-3972-4eb8-a55f-7d7c46e756da response_code = 200 uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmMzlkM2U3MC02NmU0LTQ2OGEtYTQ0ZC03NDZkMjJiZWEzYTAifX0%3D
25612025-07-17T11:32:49.676ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 8 reason = Timeout
25622025-07-17T11:32:49.676ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 8 reason = Timeout
25632025-07-17T11:32:49.676ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 10 reason = Timeout
25642025-07-17T11:32:49.676ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
25652025-07-17T11:32:49.676ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 10 reason = Timeout
25662025-07-17T11:32:49.676ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
25672025-07-17T11:32:49.676ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 15 reason = Timeout
25682025-07-17T11:32:49.676ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
25692025-07-17T11:32:49.676ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 8 reason = Timeout
25702025-07-17T11:32:49.676ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-07-17 11:22:49.676508366 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
25712025-07-17T11:32:49.676ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
25722025-07-17T11:32:49.676ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 8 reason = Timeout
25732025-07-17T11:32:49.676ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 8 reason = Timeout
25742025-07-17T11:32:49.676ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
25752025-07-17T11:32:49.676ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
25762025-07-17T11:32:49.676ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 8 reason = Timeout
25772025-07-17T11:32:49.676ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
25782025-07-17T11:32:49.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
25792025-07-17T11:32:49.677ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("3dd3bf3b-7760-4750-b1d3-52ee777335f0.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("79e5c77d-18b3-4951-8d11-cfa3eae2b3e3.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("79e5c77d-18b3-4951-8d11-cfa3eae2b3e3.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::8)) }], valid_until: Instant { tv_sec: 3763, tv_nsec: 429472976 } })
25802025-07-17T11:32:49.677ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
25812025-07-17T11:32:49.677ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
25822025-07-17T11:32:49.677ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
25832025-07-17T11:32:49.677ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
25842025-07-17T11:32:49.677ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("3144a75a-317f-4b68-93e2-a210ca1189d1.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("87408888-9df4-4e0c-bc8e-2f920d039100.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("ee4378c5-e5f2-41cf-8969-e5e03e1b9a41.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("ee4378c5-e5f2-41cf-8969-e5e03e1b9a41.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:3::1)) }], valid_until: Instant { tv_sec: 3763, tv_nsec: 429626769 } })
25852025-07-17T11:32:49.677ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
25862025-07-17T11:32:49.677ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.011793ms iteration = 8 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
25872025-07-17T11:32:49.677ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } dns_group = internal
25882025-07-17T11:32:49.677ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.210226ms iteration = 8 status = {"addresses":["[fd00:1122:3344:1::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:3::1]:5353"]}
25892025-07-17T11:32:49.678ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
25902025-07-17T11:32:49.678ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
25912025-07-17T11:32:49.679ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 9 reason = Timeout
25922025-07-17T11:32:49.679ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
25932025-07-17T11:32:49.679ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:85 reason = blueprint disabled target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a
25942025-07-17T11:32:49.679ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 17.14\u{b5}s iteration = 9 status = {"enabled":false,"target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a"}
25952025-07-17T11:32:49.679ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3763, tv_nsec: 432309074 } })
25962025-07-17T11:32:49.679ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
25972025-07-17T11:32:49.681ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1186
25982025-07-17T11:32:49.683ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-07-17 11:22:49.676508366 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
25992025-07-17T11:32:49.683ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 7.016622ms iteration = 8 status = {"expiration":"2025-07-17T11:22:49.676508366Z","pruned":[]}
26002025-07-17T11:32:49.683ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26012025-07-17T11:32:49.684ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26022025-07-17T11:32:49.684ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_planner iteration = 10 reason = Timeout
26032025-07-17T11:32:49.684ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 8 reason = Timeout
26042025-07-17T11:32:49.684ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:53
26052025-07-17T11:32:49.684ZDEBGnexus (BackgroundTasks): blueprint planning disabled, doing nothing background_task = blueprint_planner
26062025-07-17T11:32:49.684ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_planner elapsed = 22.8\u{b5}s iteration = 10 status = "Disabled"
26072025-07-17T11:32:49.684ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 8 reason = Timeout
26082025-07-17T11:32:49.684ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
26092025-07-17T11:32:49.684ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
26102025-07-17T11:32:49.685ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
26112025-07-17T11:32:49.685ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
26122025-07-17T11:32:49.685ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26132025-07-17T11:32:49.685ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 8 reason = Timeout
26142025-07-17T11:32:49.686ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
26152025-07-17T11:32:49.686ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
26162025-07-17T11:32:49.687ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
26172025-07-17T11:32:49.687ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
26182025-07-17T11:32:49.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
26192025-07-17T11:32:49.688ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
26202025-07-17T11:32:49.689ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
26212025-07-17T11:32:49.690ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26222025-07-17T11:32:49.691ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
26232025-07-17T11:32:49.691ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
26242025-07-17T11:32:49.692ZDEBGnexus (BackgroundTasks): activating background_task = webhook_deliverator iteration = 8 reason = Timeout
26252025-07-17T11:32:49.692ZDEBGnexus (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} }
26262025-07-17T11:32:49.692ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database result = Ok(())
26272025-07-17T11:32:49.692ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 8 reason = Timeout
26282025-07-17T11:32:49.692ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
26292025-07-17T11:32:49.692ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
26302025-07-17T11:32:49.692ZDEBGnexus (BackgroundTasks): activating background_task = alert_dispatcher iteration = 8 reason = Timeout
26312025-07-17T11:32:49.692ZDEBGnexus (BackgroundTasks): roles background_task = alert_dispatcher roles = RoleSet { roles: {} }
26322025-07-17T11:32:49.693ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = alert_dispatcher resource = Database result = Ok(())
26332025-07-17T11:32:49.693ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
26342025-07-17T11:32:49.693ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
26352025-07-17T11:32:49.694ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
26362025-07-17T11:32:49.694ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
26372025-07-17T11:32:49.694ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
26382025-07-17T11:32:49.694ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
26392025-07-17T11:32:49.694ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
26402025-07-17T11:32:49.694ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
26412025-07-17T11:32:49.694ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26422025-07-17T11:32:49.696ZINFOnexus (BackgroundTasks): all webhook delivery tasks completed background_task = webhook_deliverator file = nexus/src/app/background/tasks/webhook_deliverator.rs:204 num_receivers = 0
26432025-07-17T11:32:49.696ZDEBGnexus (BackgroundTasks): activation complete background_task = webhook_deliverator elapsed = 3.860341ms iteration = 8 status = {"by_rx":{},"error":null}
26442025-07-17T11:32:49.697ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
26452025-07-17T11:32:49.697ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
26462025-07-17T11:32:49.697ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
26472025-07-17T11:32:49.697ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 5.198249ms iteration = 8 status = {"errors":[],"found_port_ok":[]}
26482025-07-17T11:32:49.703ZDEBGnexus (BackgroundTasks): roles background_task = alert_dispatcher roles = RoleSet { roles: {} }
26492025-07-17T11:32:49.703ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = alert_dispatcher resource = Database result = Ok(())
26502025-07-17T11:32:49.703ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26512025-07-17T11:32:49.706ZDEBGnexus (BackgroundTasks): activation complete background_task = alert_dispatcher elapsed = 13.347726ms iteration = 8 status = {"dispatched":[],"errors":[],"glob_version":"161.0.0","globs_reprocessed":{},"no_receivers":[]}
26522025-07-17T11:32:49.708ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
26532025-07-17T11:32:49.708ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
26542025-07-17T11:32:49.708ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
26552025-07-17T11:32:49.733ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 15 reason = Timeout
26562025-07-17T11:32:49.733ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 15 reason = Timeout
26572025-07-17T11:32:49.733ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
26582025-07-17T11:32:49.733ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
26592025-07-17T11:32:49.736ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3763, tv_nsec: 489102063 } })
26602025-07-17T11:32:49.736ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1186
26612025-07-17T11:32:49.737ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.073954ms iteration = 15 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
26622025-07-17T11:32:49.750ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
26632025-07-17T11:32:49.750ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:170 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
26642025-07-17T11:32:49.750ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 64.955916ms iteration = 8 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
26652025-07-17T11:32:49.763ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 15 reason = Timeout
26662025-07-17T11:32:49.763ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
26672025-07-17T11:32:49.763ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 15 reason = Timeout
26682025-07-17T11:32:49.763ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 15 reason = Timeout
26692025-07-17T11:32:49.763ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
26702025-07-17T11:32:49.763ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
26712025-07-17T11:32:49.763ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
26722025-07-17T11:32:49.763ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 15 reason = Timeout
26732025-07-17T11:32:49.763ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
26742025-07-17T11:32:49.763ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 15 reason = Timeout
26752025-07-17T11:32:49.763ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 15 reason = Timeout
26762025-07-17T11:32:49.763ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
26772025-07-17T11:32:49.763ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 15 reason = Timeout
26782025-07-17T11:32:49.763ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
26792025-07-17T11:32:49.763ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
26802025-07-17T11:32:49.763ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 15 reason = Timeout
26812025-07-17T11:32:49.763ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 15 reason = Timeout
26822025-07-17T11:32:49.763ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
26832025-07-17T11:32:49.763ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
26842025-07-17T11:32:49.763ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 15 reason = Timeout
26852025-07-17T11:32:49.763ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
26862025-07-17T11:32:49.763ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
26872025-07-17T11:32:49.763ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 43 reason = Timeout
26882025-07-17T11:32:49.763ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 15 reason = Timeout
26892025-07-17T11:32:49.764ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
26902025-07-17T11:32:49.764ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
26912025-07-17T11:32:49.764ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 43 reason = Timeout
26922025-07-17T11:32:49.764ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 15 reason = Timeout
26932025-07-17T11:32:49.764ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
26942025-07-17T11:32:49.764ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
26952025-07-17T11:32:49.764ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 16 reason = Timeout
26962025-07-17T11:32:49.764ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
26972025-07-17T11:32:49.764ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 85 reason = Timeout
26982025-07-17T11:32:49.764ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
26992025-07-17T11:32:49.764ZDEBGnexus (BackgroundTasks): activating background_task = sp_ereport_ingester iteration = 15 reason = Timeout
27002025-07-17T11:32:49.764ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 15 reason = Timeout
27012025-07-17T11:32:49.764ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
27022025-07-17T11:32:49.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
27032025-07-17T11:32:49.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
27042025-07-17T11:32:49.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
27052025-07-17T11:32:49.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
27062025-07-17T11:32:49.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
27072025-07-17T11:32:49.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
27082025-07-17T11:32:49.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
27092025-07-17T11:32:49.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
27102025-07-17T11:32:49.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
27112025-07-17T11:32:49.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
27122025-07-17T11:32:49.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
27132025-07-17T11:32:49.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
27142025-07-17T11:32:49.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
27152025-07-17T11:32:49.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
27162025-07-17T11:32:49.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
27172025-07-17T11:32:49.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
27182025-07-17T11:32:49.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
27192025-07-17T11:32:49.764ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12225, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3763, tv_nsec: 517204223 } })
27202025-07-17T11:32:49.766ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
27212025-07-17T11:32:49.766ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
27222025-07-17T11:32:49.766ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
27232025-07-17T11:32:49.766ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
27242025-07-17T11:32:49.766ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
27252025-07-17T11:32:49.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
27262025-07-17T11:32:49.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
27272025-07-17T11:32:49.767ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 4.131565ms iteration = 15 status = {"errors":[],"finish_invoked_ok":[]}
27282025-07-17T11:32:49.768ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
27292025-07-17T11:32:49.768ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
27302025-07-17T11:32:49.768ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
27312025-07-17T11:32:49.768ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27322025-07-17T11:32:49.768ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
27332025-07-17T11:32:49.769ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27342025-07-17T11:32:49.769ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 6.095721ms iteration = 15 status = {"errors":[],"garbage_collect_requested":[]}
27352025-07-17T11:32:49.770ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
27362025-07-17T11:32:49.770ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 915fbba0-947f-4bf0-9f9f-48a79df81f7b dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
27372025-07-17T11:32:49.770ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
27382025-07-17T11:32:49.770ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27392025-07-17T11:32:49.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
27402025-07-17T11:32:49.770ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
27412025-07-17T11:32:49.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
27422025-07-17T11:32:49.771ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
27432025-07-17T11:32:49.771ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
27442025-07-17T11:32:49.771ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27452025-07-17T11:32:49.771ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
27462025-07-17T11:32:49.771ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:138 found = 0 not_ready_to_be_deleted = 0
27472025-07-17T11:32:49.771ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 87.030477ms iteration = 8 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
27482025-07-17T11:32:49.772ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27492025-07-17T11:32:49.772ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27502025-07-17T11:32:49.772ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
27512025-07-17T11:32:49.772ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
27522025-07-17T11:32:49.773ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27532025-07-17T11:32:49.773ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
27542025-07-17T11:32:49.773ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
27552025-07-17T11:32:49.773ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
27562025-07-17T11:32:49.773ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
27572025-07-17T11:32:49.773ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27582025-07-17T11:32:49.774ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27592025-07-17T11:32:49.774ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 10.682911ms iteration = 15 status = {"errors":[],"requests_created_ok":[]}
27602025-07-17T11:32:49.775ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
27612025-07-17T11:32:49.775ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
27622025-07-17T11:32:49.775ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
27632025-07-17T11:32:49.775ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
27642025-07-17T11:32:49.775ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
27652025-07-17T11:32:49.776ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27662025-07-17T11:32:49.776ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
27672025-07-17T11:32:49.776ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
27682025-07-17T11:32:49.776ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
27692025-07-17T11:32:49.776ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27702025-07-17T11:32:49.776ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
27712025-07-17T11:32:49.776ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 12.93966ms iteration = 43 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
27722025-07-17T11:32:49.777ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
27732025-07-17T11:32:49.777ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
27742025-07-17T11:32:49.777ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
27752025-07-17T11:32:49.777ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
27762025-07-17T11:32:49.777ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
27772025-07-17T11:32:49.777ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
27782025-07-17T11:32:49.778ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
27792025-07-17T11:32:49.778ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
27802025-07-17T11:32:49.778ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
27812025-07-17T11:32:49.778ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
27822025-07-17T11:32:49.778ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
27832025-07-17T11:32:49.779ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
27842025-07-17T11:32:49.779ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
27852025-07-17T11:32:49.779ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
27862025-07-17T11:32:49.779ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
27872025-07-17T11:32:49.779ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
27882025-07-17T11:32:49.779ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27892025-07-17T11:32:49.779ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
27902025-07-17T11:32:49.781ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3763, tv_nsec: 533594999 } })
27912025-07-17T11:32:49.781ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1186 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
27922025-07-17T11:32:49.781ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
27932025-07-17T11:32:49.781ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
27942025-07-17T11:32:49.781ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
27952025-07-17T11:32:49.781ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
27962025-07-17T11:32:49.781ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
27972025-07-17T11:32:49.782ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
27982025-07-17T11:32:49.782ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
27992025-07-17T11:32:49.782ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
28002025-07-17T11:32:49.782ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 18.19665ms iteration = 85 status = {"chicken_switches_updated":false}
28012025-07-17T11:32:49.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
28022025-07-17T11:32:49.782ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
28032025-07-17T11:32:49.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
28042025-07-17T11:32:49.782ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:550 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
28052025-07-17T11:32:49.782ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 19.071631ms iteration = 15 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
28062025-07-17T11:32:49.783ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
28072025-07-17T11:32:49.783ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28082025-07-17T11:32:49.783ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
28092025-07-17T11:32:49.783ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
28102025-07-17T11:32:49.783ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
28112025-07-17T11:32:49.783ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
28122025-07-17T11:32:49.783ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
28132025-07-17T11:32:49.785ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
28142025-07-17T11:32:49.785ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
28152025-07-17T11:32:49.786ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
28162025-07-17T11:32:49.787ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 915fbba0-947f-4bf0-9f9f-48a79df81f7b dns_zone_name = control-plane.oxide.internal found_so_far = 59 version = 1
28172025-07-17T11:32:49.787ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
28182025-07-17T11:32:49.787ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-07-17 11:26:34.922237 UTC dns_group = internal generation = 1
28192025-07-17T11:32:49.787ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 110.73494ms iteration = 8 status = {"generation":1}
28202025-07-17T11:32:49.787ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
28212025-07-17T11:32:49.788ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
28222025-07-17T11:32:49.800ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
28232025-07-17T11:32:49.800ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 36.547322ms iteration = 43 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
28242025-07-17T11:32:49.839ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
28252025-07-17T11:32:49.839ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
28262025-07-17T11:32:49.843ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
28272025-07-17T11:32:49.843ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
28282025-07-17T11:32:49.858ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28292025-07-17T11:32:49.859ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
28302025-07-17T11:32:49.859ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
28312025-07-17T11:32:49.859ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 95.898734ms iteration = 15 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
28322025-07-17T11:32:49.860ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
28332025-07-17T11:32:49.861ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
28342025-07-17T11:32:49.863ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
28352025-07-17T11:32:49.863ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
28362025-07-17T11:32:49.863ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
28372025-07-17T11:32:49.864ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28382025-07-17T11:32:49.864ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
28392025-07-17T11:32:49.864ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 4a9e4c5e-728e-4850-bd58-14548c02bf6f dns_zone_name = oxide.test found_so_far = 0 version = 2
28402025-07-17T11:32:49.864ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
28412025-07-17T11:32:49.865ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
28422025-07-17T11:32:49.865ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
28432025-07-17T11:32:49.865ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
28442025-07-17T11:32:49.866ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 102.286819ms iteration = 15 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
28452025-07-17T11:32:49.867ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28462025-07-17T11:32:49.869ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
28472025-07-17T11:32:49.869ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
28482025-07-17T11:32:49.870ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-07-17T11:25:46.964149Z, time_modified: 2025-07-17T11:25:46.964149Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 1fa836d9-7854-4ada-8c89-c26d922646c6, db_silo: Silo { identity: SiloIdentity { id: 1fa836d9-7854-4ada-8c89-c26d922646c6, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-07-17T11:26:35.643175Z, time_modified: 2025-07-17T11:26:35.643175Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "d7c3455b70a66c86ec2a7e597ca46fe0ea0c49dde5c1363f198c20aa77d58442" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 1fa836d9-7854-4ada-8c89-c26d922646c6, db_silo: Silo { identity: SiloIdentity { id: 1fa836d9-7854-4ada-8c89-c26d922646c6, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-07-17T11:26:35.643175Z, time_modified: 2025-07-17T11:26:35.643175Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "d7c3455b70a66c86ec2a7e597ca46fe0ea0c49dde5c1363f198c20aa77d58442" }] }) }
28492025-07-17T11:32:49.870ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 193.690614ms iteration = 8 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"1fa836d9-7854-4ada-8c89-c26d922646c6","tls_certs":["d7c3455b70a66c86ec2a7e597ca46fe0ea0c49dde5c1363f198c20aa77d58442"]}},"default_endpoint":{"silo_id":"1fa836d9-7854-4ada-8c89-c26d922646c6","tls_certs":["d7c3455b70a66c86ec2a7e597ca46fe0ea0c49dde5c1363f198c20aa77d58442"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
28502025-07-17T11:32:49.870ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
28512025-07-17T11:32:49.870ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
28522025-07-17T11:32:49.870ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
28532025-07-17T11:32:49.871ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
28542025-07-17T11:32:49.871ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28552025-07-17T11:32:49.873ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:372
28562025-07-17T11:32:49.873ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 109.460013ms iteration = 15 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
28572025-07-17T11:32:49.874ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
28582025-07-17T11:32:49.874ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
28592025-07-17T11:32:49.874ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
28602025-07-17T11:32:49.877ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 0ae78f24-48db-4cf7-91cb-8d5311f2a17f, lookup_type: ById(0ae78f24-48db-4cf7-91cb-8d5311f2a17f) } result = Ok(())
28612025-07-17T11:32:49.877ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
28622025-07-17T11:32:49.877ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
28632025-07-17T11:32:49.877ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 4a9e4c5e-728e-4850-bd58-14548c02bf6f dns_zone_name = oxide.test found_so_far = 4 version = 2
28642025-07-17T11:32:49.877ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
28652025-07-17T11:32:49.877ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-07-17 11:26:35.847594 UTC dns_group = external generation = 2
28662025-07-17T11:32:49.877ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 201.043621ms iteration = 8 status = {"generation":2}
28672025-07-17T11:32:49.879ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 115.986969ms iteration = 15 status = 0
28682025-07-17T11:32:49.892ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
28692025-07-17T11:32:49.893ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
28702025-07-17T11:32:49.908ZINFOnexus (BackgroundTasks): instance reincarnation completed successfully background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:113 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0
28712025-07-17T11:32:49.908ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 223.294784ms iteration = 8 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
28722025-07-17T11:32:49.935ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 171.242507ms iteration = 15 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
28732025-07-17T11:32:49.937ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1194 zone_address = fd00:1122:3344:101::2
28742025-07-17T11:32:49.937ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
28752025-07-17T11:32:49.937ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "c01a5109-0e89-4a3f-a995-cd964fd44004", "content-length": "26", "date": "Thu, 17 Jul 2025 11:32:49 GMT"} })
28762025-07-17T11:32:49.937ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1197 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
28772025-07-17T11:32:49.938ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 174.323568ms iteration = 15 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
28782025-07-17T11:32:49.939ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
28792025-07-17T11:32:50.023ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
28802025-07-17T11:32:50.024ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
28812025-07-17T11:32:50.025ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1194 zone_address = fd00:1122:3344:101::2
28822025-07-17T11:32:50.025ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
28832025-07-17T11:32:50.026ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28842025-07-17T11:32:50.026ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d5f84c8-a970-487b-9df7-abeffe3f2361", "content-length": "26", "date": "Thu, 17 Jul 2025 11:32:49 GMT"} })
28852025-07-17T11:32:50.026ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1197 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
28862025-07-17T11:32:50.027ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
28872025-07-17T11:32:50.029ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
28882025-07-17T11:32:50.029ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
28892025-07-17T11:32:50.029ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
28902025-07-17T11:32:50.031ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
28912025-07-17T11:32:50.034ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
28922025-07-17T11:32:50.034ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
28932025-07-17T11:32:50.034ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
28942025-07-17T11:32:50.075ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 311.220553ms iteration = 16 status = {"physical_disks_added":0}
28952025-07-17T11:32:50.099ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
28962025-07-17T11:32:50.143ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/all-sp-ids
28972025-07-17T11:32:50.143ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/all-sp-ids", status: 200, headers: {"content-type": "application/json", "x-request-id": "2cbf2c51-eb3a-48d4-8c71-cf5b5c07d9b3", "content-length": "78", "date": "Thu, 17 Jul 2025 11:32:49 GMT"} })
28982025-07-17T11:32:50.143ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = switch
28992025-07-17T11:32:50.144ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = sled
29002025-07-17T11:32:50.144ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 1 sp_type = sled
29012025-07-17T11:32:50.144ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = switch
29022025-07-17T11:32:50.144ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = sled
29032025-07-17T11:32:50.144ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 1 sp_type = sled
29042025-07-17T11:32:50.146ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 0 sp_type = switch
29052025-07-17T11:32:50.146ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 0 sp_type = sled
29062025-07-17T11:32:50.147ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 1 sp_type = sled
29072025-07-17T11:32:50.150ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 0 sp_type = sled
29082025-07-17T11:32:50.150ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = sled
29092025-07-17T11:32:50.150ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 0 sp_type = switch
29102025-07-17T11:32:50.150ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = switch
29112025-07-17T11:32:50.150ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = sled
29122025-07-17T11:32:50.150ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 1 sp_type = sled
29132025-07-17T11:32:50.150ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 1 sp_type = sled
29142025-07-17T11:32:50.150ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = switch
29152025-07-17T11:32:50.150ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 1 sp_type = sled
29162025-07-17T11:32:50.152ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 0 sp_type = sled start_ena = None
29172025-07-17T11:32:50.152ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
29182025-07-17T11:32:50.152ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "39c10d41-b0ea-4c0d-8eb9-81b0b0d40e8c", "content-length": "81", "date": "Thu, 17 Jul 2025 11:32:49 GMT"} })
29192025-07-17T11:32:50.153ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 0 sp_type = switch start_ena = None
29202025-07-17T11:32:50.153ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 1 sp_type = sled start_ena = None
29212025-07-17T11:32:50.153ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
29222025-07-17T11:32:50.153ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/1/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
29232025-07-17T11:32:50.153ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "7fb8ec8a-77e1-47b2-9077-211927965dcb", "content-length": "81", "date": "Thu, 17 Jul 2025 11:32:49 GMT"} })
29242025-07-17T11:32:50.153ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/1/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf18b73f-ad58-46ab-afaf-34ba980a3024", "content-length": "81", "date": "Thu, 17 Jul 2025 11:32:50 GMT"} })
29252025-07-17T11:32:50.155ZDEBGnexus (BackgroundTasks): activation complete background_task = sp_ereport_ingester elapsed = 390.985275ms iteration = 15 status = {"errors":[],"sps":[]}
29262025-07-17T11:32:50.186ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1194 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 zone_address = fd00:1122:3344:101::2
29272025-07-17T11:32:50.186ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
29282025-07-17T11:32:50.187ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "9247e62b-8bf1-4f68-961f-0ad96ee1fd3c", "content-length": "26", "date": "Thu, 17 Jul 2025 11:32:50 GMT"} })
29292025-07-17T11:32:50.187ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1197 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
29302025-07-17T11:32:50.189ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
29312025-07-17T11:32:50.247ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
29322025-07-17T11:32:50.247ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "d4c6c58f-fd36-4b2d-841f-002f365aa1e1", "date": "Thu, 17 Jul 2025 11:32:50 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
29332025-07-17T11:32:50.247ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
29342025-07-17T11:32:50.248ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "ad2468c6-81a2-4f44-850d-eaf78c946e0f", "date": "Thu, 17 Jul 2025 11:32:50 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
29352025-07-17T11:32:50.248ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
29362025-07-17T11:32:50.250ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
29372025-07-17T11:32:50.250ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 574.069869ms iteration = 10 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
29382025-07-17T11:32:50.290ZINFOnexus (BackgroundTasks): VPC route manager sled 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
29392025-07-17T11:32:50.290ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
29402025-07-17T11:32:50.290ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d79e0c7-7435-4738-b3ec-60f56b93dcdb", "content-length": "372", "date": "Thu, 17 Jul 2025 11:32:50 GMT"} })
29412025-07-17T11:32:50.290ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
29422025-07-17T11:32:50.291ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
29432025-07-17T11:32:50.311ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
29442025-07-17T11:32:50.311ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
29452025-07-17T11:32:50.311ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "d43b428c-fbeb-4b45-b5d2-497f4243e5a1", "date": "Thu, 17 Jul 2025 11:32:50 GMT"} })
29462025-07-17T11:32:50.311ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
29472025-07-17T11:32:50.311ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
29482025-07-17T11:32:50.314ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
29492025-07-17T11:32:50.314ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 580.943529ms iteration = 15 status = {}
29502025-07-17T11:32:50.314ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
29512025-07-17T11:32:50.316ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
29522025-07-17T11:32:50.317ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
29532025-07-17T11:32:50.318ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
29542025-07-17T11:32:50.318ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
29552025-07-17T11:32:50.319ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 642.930616ms iteration = 15 status = null
29562025-07-17T11:32:50.321ZINFOnexus (BackgroundTasks): VPC route manager sled 0ae78f24-48db-4cf7-91cb-8d5311f2a17f push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
29572025-07-17T11:32:50.321ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
29582025-07-17T11:32:50.322ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "48d251ee-1936-4376-a9c8-0fbe1e8f655b", "date": "Thu, 17 Jul 2025 11:32:50 GMT"} })
29592025-07-17T11:32:50.323ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 559.631298ms iteration = 15 status = {}
29602025-07-17T11:32:50.340ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
29612025-07-17T11:32:50.341ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "82a06ad8-fd9c-4b98-89c0-b85759a51877", "content-length": "691", "date": "Thu, 17 Jul 2025 11:32:50 GMT"} })
29622025-07-17T11:32:50.341ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
29632025-07-17T11:32:50.341ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
29642025-07-17T11:32:50.341ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
29652025-07-17T11:32:50.342ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 578.662489ms iteration = 15 status = {}
29662025-07-17T11:32:50.355ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
29672025-07-17T11:32:50.478ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
29682025-07-17T11:32:50.479ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "1e8cb012-24b3-45b6-9cd3-bafca374f159", "date": "Thu, 17 Jul 2025 11:32:50 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
29692025-07-17T11:32:50.479ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
29702025-07-17T11:32:50.481ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "9a1c1344-1d86-4763-859f-7bb32e24ada0", "date": "Thu, 17 Jul 2025 11:32:50 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
29712025-07-17T11:32:50.481ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
29722025-07-17T11:32:50.483ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "ac60da49-bd2b-41c4-8a7e-7e230e747a55", "date": "Thu, 17 Jul 2025 11:32:50 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
29732025-07-17T11:32:50.484ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
29742025-07-17T11:32:50.486ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
29752025-07-17T11:32:50.486ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 810.008299ms iteration = 10 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
29762025-07-17T11:32:50.520ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
29772025-07-17T11:32:50.520ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
29782025-07-17T11:32:50.522ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 151caa7c-6a05-4995-94b4-7946d428117d, rack_id: 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5, switch_location: "switch0", port_name: Name(Name("qsfp0")), port_settings_id: Some(2e30c7ae-26da-46e4-9127-aac194921f20) } rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0
29792025-07-17T11:32:50.522ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
29802025-07-17T11:32:50.523ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
29812025-07-17T11:32:50.537ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1832 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = Name(Name("qsfp0"))
29822025-07-17T11:32:50.537ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1859 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0 switch_port_id = Name(Name("qsfp0")) switch_port_settings_id = 2e30c7ae-26da-46e4-9127-aac194921f20
29832025-07-17T11:32:50.537ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
29842025-07-17T11:32:50.537ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
29852025-07-17T11:32:50.537ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
29862025-07-17T11:32:50.537ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
29872025-07-17T11:32:50.537ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
29882025-07-17T11:32:50.537ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2006 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
29892025-07-17T11:32:50.537ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
29902025-07-17T11:32:50.537ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2042 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
29912025-07-17T11:32:50.537ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
29922025-07-17T11:32:50.537ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
29932025-07-17T11:32:50.538ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
29942025-07-17T11:32:50.540ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0
29952025-07-17T11:32:50.540ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
29962025-07-17T11:32:50.721ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "eaaaaa89-4089-4841-94a8-916f690238c3", "date": "Thu, 17 Jul 2025 11:32:50 GMT"} })
29972025-07-17T11:32:50.721ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch = Switch0
29982025-07-17T11:32:50.721ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
29992025-07-17T11:32:50.721ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7e9e0dc-b571-4329-a65d-ef043f41bf96", "content-length": "682", "date": "Thu, 17 Jul 2025 11:32:50 GMT"} })
30002025-07-17T11:32:50.722ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
30012025-07-17T11:32:50.722ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
30022025-07-17T11:32:50.726ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
30032025-07-17T11:32:50.726ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
30042025-07-17T11:32:50.730ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
30052025-07-17T11:32:50.730ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
30062025-07-17T11:32:50.732ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
30072025-07-17T11:32:50.732ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
30082025-07-17T11:32:50.738ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 974.893963ms iteration = 15 status = {}
30092025-07-17T11:32:51.692ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 317 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = 1c3371a9-5f4f-414e-9697-77a3436d8baa response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
30102025-07-17T11:32:54.764ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 86 reason = Timeout
30112025-07-17T11:32:54.764ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
30122025-07-17T11:32:54.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
30132025-07-17T11:32:54.766ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30142025-07-17T11:32:54.769ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
30152025-07-17T11:32:54.769ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
30162025-07-17T11:32:54.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
30172025-07-17T11:32:54.771ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
30182025-07-17T11:32:54.771ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 7.043433ms iteration = 86 status = {"chicken_switches_updated":false}
30192025-07-17T11:32:59.764ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 44 reason = Timeout
30202025-07-17T11:32:59.764ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
30212025-07-17T11:32:59.764ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 44 reason = Timeout
30222025-07-17T11:32:59.764ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 87 reason = Timeout
30232025-07-17T11:32:59.764ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
30242025-07-17T11:32:59.764ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
30252025-07-17T11:32:59.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
30262025-07-17T11:32:59.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
30272025-07-17T11:32:59.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
30282025-07-17T11:32:59.766ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
30292025-07-17T11:32:59.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
30302025-07-17T11:32:59.766ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30312025-07-17T11:32:59.767ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30322025-07-17T11:32:59.768ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.749849ms iteration = 44 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
30332025-07-17T11:32:59.770ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
30342025-07-17T11:32:59.770ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
30352025-07-17T11:32:59.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
30362025-07-17T11:32:59.770ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
30372025-07-17T11:32:59.770ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
30382025-07-17T11:32:59.771ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
30392025-07-17T11:32:59.771ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
30402025-07-17T11:32:59.771ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
30412025-07-17T11:32:59.772ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
30422025-07-17T11:32:59.772ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 7.997455ms iteration = 87 status = {"chicken_switches_updated":false}
30432025-07-17T11:32:59.772ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30442025-07-17T11:32:59.775ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
30452025-07-17T11:32:59.775ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
30462025-07-17T11:32:59.775ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
30472025-07-17T11:32:59.785ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
30482025-07-17T11:32:59.785ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.786944ms iteration = 44 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
30492025-07-17T11:33:01.683ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:35131 req_id = 80e19b7c-f24a-4086-a425-8355c2a62414 roles = RoleSet { roles: {} } uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?sort_by=id_ascending
30502025-07-17T11:33:01.684ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:35131 req_id = 80e19b7c-f24a-4086-a425-8355c2a62414 resource = Database result = Ok(()) uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?sort_by=id_ascending
30512025-07-17T11:33:01.685ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 2080 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:35131 req_id = 80e19b7c-f24a-4086-a425-8355c2a62414 response_code = 200 uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?sort_by=id_ascending
30522025-07-17T11:33:01.686ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:35131 req_id = 0be9bd92-66c7-47c3-acef-3dc6ae72f07f roles = RoleSet { roles: {} } uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmMzlkM2U3MC02NmU0LTQ2OGEtYTQ0ZC03NDZkMjJiZWEzYTAifX0%3D
30532025-07-17T11:33:01.686ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:35131 req_id = 0be9bd92-66c7-47c3-acef-3dc6ae72f07f resource = Database result = Ok(()) uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmMzlkM2U3MC02NmU0LTQ2OGEtYTQ0ZC03NDZkMjJiZWEzYTAifX0%3D
30542025-07-17T11:33:01.687ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 1794 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:35131 req_id = 0be9bd92-66c7-47c3-acef-3dc6ae72f07f response_code = 200 uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmMzlkM2U3MC02NmU0LTQ2OGEtYTQ0ZC03NDZkMjJiZWEzYTAifX0%3D
30552025-07-17T11:33:01.692ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 365 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = 0bec7d38-b992-4aec-ad37-e042a2449eec response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
30562025-07-17T11:33:04.763ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 88 reason = Timeout
30572025-07-17T11:33:04.763ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
30582025-07-17T11:33:04.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
30592025-07-17T11:33:04.766ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30602025-07-17T11:33:04.769ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
30612025-07-17T11:33:04.769ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
30622025-07-17T11:33:04.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
30632025-07-17T11:33:04.770ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
30642025-07-17T11:33:04.770ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 6.992302ms iteration = 88 status = {"chicken_switches_updated":false}
30652025-07-17T11:33:09.764ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 45 reason = Timeout
30662025-07-17T11:33:09.764ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
30672025-07-17T11:33:09.764ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 45 reason = Timeout
30682025-07-17T11:33:09.764ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
30692025-07-17T11:33:09.764ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 89 reason = Timeout
30702025-07-17T11:33:09.764ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
30712025-07-17T11:33:09.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
30722025-07-17T11:33:09.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
30732025-07-17T11:33:09.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
30742025-07-17T11:33:09.766ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
30752025-07-17T11:33:09.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
30762025-07-17T11:33:09.766ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30772025-07-17T11:33:09.767ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30782025-07-17T11:33:09.768ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.408548ms iteration = 45 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
30792025-07-17T11:33:09.770ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
30802025-07-17T11:33:09.770ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
30812025-07-17T11:33:09.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
30822025-07-17T11:33:09.770ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
30832025-07-17T11:33:09.770ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
30842025-07-17T11:33:09.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
30852025-07-17T11:33:09.772ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
30862025-07-17T11:33:09.772ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
30872025-07-17T11:33:09.772ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 8.663994ms iteration = 89 status = {"chicken_switches_updated":false}
30882025-07-17T11:33:09.772ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
30892025-07-17T11:33:09.774ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
30902025-07-17T11:33:09.776ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
30912025-07-17T11:33:09.776ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
30922025-07-17T11:33:09.777ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
30932025-07-17T11:33:09.786ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
30942025-07-17T11:33:09.786ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.937213ms iteration = 45 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
30952025-07-17T11:33:11.692ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 381 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = 0ebfdd11-f560-409e-94eb-c2e346fc4ae7 response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
30962025-07-17T11:33:14.763ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 90 reason = Timeout
30972025-07-17T11:33:14.764ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
30982025-07-17T11:33:14.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
30992025-07-17T11:33:14.766ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31002025-07-17T11:33:14.769ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
31012025-07-17T11:33:14.769ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
31022025-07-17T11:33:14.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
31032025-07-17T11:33:14.770ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
31042025-07-17T11:33:14.770ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 6.580407ms iteration = 90 status = {"chicken_switches_updated":false}
31052025-07-17T11:33:19.676ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 16 reason = Timeout
31062025-07-17T11:33:19.676ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
31072025-07-17T11:33:19.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
31082025-07-17T11:33:19.678ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3793, tv_nsec: 430691159 } })
31092025-07-17T11:33:19.678ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1186
31102025-07-17T11:33:19.708ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1194 zone_address = fd00:1122:3344:101::2
31112025-07-17T11:33:19.708ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
31122025-07-17T11:33:19.709ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "7746dd29-370a-4791-9968-9d478c4ff530", "content-length": "26", "date": "Thu, 17 Jul 2025 11:33:19 GMT"} })
31132025-07-17T11:33:19.709ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1197 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
31142025-07-17T11:33:19.709ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
31152025-07-17T11:33:19.733ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 16 reason = Timeout
31162025-07-17T11:33:19.733ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 16 reason = Timeout
31172025-07-17T11:33:19.733ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
31182025-07-17T11:33:19.734ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
31192025-07-17T11:33:19.735ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 2.034377ms iteration = 16 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
31202025-07-17T11:33:19.736ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3793, tv_nsec: 488724584 } })
31212025-07-17T11:33:19.736ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1186
31222025-07-17T11:33:19.743ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
31232025-07-17T11:33:19.744ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
31242025-07-17T11:33:19.745ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 69.901262ms iteration = 16 status = null
31252025-07-17T11:33:19.764ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 16 reason = Timeout
31262025-07-17T11:33:19.764ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 16 reason = Timeout
31272025-07-17T11:33:19.764ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
31282025-07-17T11:33:19.764ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
31292025-07-17T11:33:19.764ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 16 reason = Timeout
31302025-07-17T11:33:19.764ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
31312025-07-17T11:33:19.764ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 46 reason = Timeout
31322025-07-17T11:33:19.764ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 46 reason = Timeout
31332025-07-17T11:33:19.764ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
31342025-07-17T11:33:19.764ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 91 reason = Timeout
31352025-07-17T11:33:19.764ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
31362025-07-17T11:33:19.764ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
31372025-07-17T11:33:19.764ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 16 reason = Timeout
31382025-07-17T11:33:19.764ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
31392025-07-17T11:33:19.764ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 16 reason = Timeout
31402025-07-17T11:33:19.764ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
31412025-07-17T11:33:19.764ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 16 reason = Timeout
31422025-07-17T11:33:19.764ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
31432025-07-17T11:33:19.764ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
31442025-07-17T11:33:19.764ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 16 reason = Timeout
31452025-07-17T11:33:19.764ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 16 reason = Timeout
31462025-07-17T11:33:19.764ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
31472025-07-17T11:33:19.764ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 16 reason = Timeout
31482025-07-17T11:33:19.764ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
31492025-07-17T11:33:19.764ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
31502025-07-17T11:33:19.764ZDEBGnexus (BackgroundTasks): activating background_task = sp_ereport_ingester iteration = 16 reason = Timeout
31512025-07-17T11:33:19.764ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 17 reason = Timeout
31522025-07-17T11:33:19.764ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
31532025-07-17T11:33:19.764ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 16 reason = Timeout
31542025-07-17T11:33:19.764ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
31552025-07-17T11:33:19.764ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
31562025-07-17T11:33:19.764ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 16 reason = Timeout
31572025-07-17T11:33:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
31582025-07-17T11:33:19.764ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
31592025-07-17T11:33:19.764ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 16 reason = Timeout
31602025-07-17T11:33:19.764ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 16 reason = Timeout
31612025-07-17T11:33:19.764ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
31622025-07-17T11:33:19.764ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
31632025-07-17T11:33:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
31642025-07-17T11:33:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
31652025-07-17T11:33:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
31662025-07-17T11:33:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
31672025-07-17T11:33:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
31682025-07-17T11:33:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
31692025-07-17T11:33:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
31702025-07-17T11:33:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
31712025-07-17T11:33:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
31722025-07-17T11:33:19.764ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12225, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3793, tv_nsec: 517108889 } })
31732025-07-17T11:33:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
31742025-07-17T11:33:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
31752025-07-17T11:33:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
31762025-07-17T11:33:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
31772025-07-17T11:33:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
31782025-07-17T11:33:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
31792025-07-17T11:33:19.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
31802025-07-17T11:33:19.766ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31812025-07-17T11:33:19.768ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 4.500859ms iteration = 16 status = {"errors":[],"garbage_collect_requested":[]}
31822025-07-17T11:33:19.769ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 5.509213ms iteration = 16 status = {"errors":[],"finish_invoked_ok":[]}
31832025-07-17T11:33:19.769ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
31842025-07-17T11:33:19.769ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
31852025-07-17T11:33:19.769ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31862025-07-17T11:33:19.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
31872025-07-17T11:33:19.770ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
31882025-07-17T11:33:19.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
31892025-07-17T11:33:19.770ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31902025-07-17T11:33:19.771ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31912025-07-17T11:33:19.771ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31922025-07-17T11:33:19.771ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31932025-07-17T11:33:19.772ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
31942025-07-17T11:33:19.773ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31952025-07-17T11:33:19.773ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
31962025-07-17T11:33:19.773ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
31972025-07-17T11:33:19.774ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
31982025-07-17T11:33:19.774ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
31992025-07-17T11:33:19.774ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
32002025-07-17T11:33:19.774ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 10.506969ms iteration = 91 status = {"chicken_switches_updated":false}
32012025-07-17T11:33:19.774ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
32022025-07-17T11:33:19.775ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 10.901914ms iteration = 16 status = {"errors":[],"requests_created_ok":[]}
32032025-07-17T11:33:19.775ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
32042025-07-17T11:33:19.775ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
32052025-07-17T11:33:19.775ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
32062025-07-17T11:33:19.775ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 11.630494ms iteration = 16 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
32072025-07-17T11:33:19.775ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
32082025-07-17T11:33:19.775ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
32092025-07-17T11:33:19.775ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
32102025-07-17T11:33:19.775ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
32112025-07-17T11:33:19.776ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
32122025-07-17T11:33:19.776ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
32132025-07-17T11:33:19.776ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
32142025-07-17T11:33:19.776ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
32152025-07-17T11:33:19.776ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
32162025-07-17T11:33:19.776ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
32172025-07-17T11:33:19.777ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
32182025-07-17T11:33:19.777ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
32192025-07-17T11:33:19.777ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3793, tv_nsec: 529582813 } })
32202025-07-17T11:33:19.777ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
32212025-07-17T11:33:19.777ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
32222025-07-17T11:33:19.777ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1186 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
32232025-07-17T11:33:19.777ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
32242025-07-17T11:33:19.778ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
32252025-07-17T11:33:19.778ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32262025-07-17T11:33:19.778ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32272025-07-17T11:33:19.778ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
32282025-07-17T11:33:19.778ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
32292025-07-17T11:33:19.778ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
32302025-07-17T11:33:19.778ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
32312025-07-17T11:33:19.779ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 15.066049ms iteration = 16 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
32322025-07-17T11:33:19.780ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
32332025-07-17T11:33:19.780ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
32342025-07-17T11:33:19.780ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
32352025-07-17T11:33:19.780ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
32362025-07-17T11:33:19.781ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32372025-07-17T11:33:19.781ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
32382025-07-17T11:33:19.782ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32392025-07-17T11:33:19.783ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
32402025-07-17T11:33:19.783ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
32412025-07-17T11:33:19.783ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
32422025-07-17T11:33:19.783ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1194 zone_address = fd00:1122:3344:101::2
32432025-07-17T11:33:19.783ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
32442025-07-17T11:33:19.784ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
32452025-07-17T11:33:19.784ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
32462025-07-17T11:33:19.784ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "a5bb04d1-f8b9-47dc-8cf4-308295b2edb4", "content-length": "26", "date": "Thu, 17 Jul 2025 11:33:19 GMT"} })
32472025-07-17T11:33:19.784ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1197 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
32482025-07-17T11:33:19.784ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
32492025-07-17T11:33:19.784ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32502025-07-17T11:33:19.785ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32512025-07-17T11:33:19.785ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
32522025-07-17T11:33:19.785ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
32532025-07-17T11:33:19.785ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
32542025-07-17T11:33:19.786ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32552025-07-17T11:33:19.788ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
32562025-07-17T11:33:19.788ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
32572025-07-17T11:33:19.788ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
32582025-07-17T11:33:19.788ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32592025-07-17T11:33:19.788ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
32602025-07-17T11:33:19.788ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
32612025-07-17T11:33:19.789ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:550 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
32622025-07-17T11:33:19.789ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 24.985649ms iteration = 16 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
32632025-07-17T11:33:19.789ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
32642025-07-17T11:33:19.789ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
32652025-07-17T11:33:19.789ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
32662025-07-17T11:33:19.791ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32672025-07-17T11:33:19.794ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
32682025-07-17T11:33:19.794ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
32692025-07-17T11:33:19.794ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
32702025-07-17T11:33:19.802ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
32712025-07-17T11:33:19.802ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 38.569108ms iteration = 46 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
32722025-07-17T11:33:19.822ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
32732025-07-17T11:33:19.822ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
32742025-07-17T11:33:19.822ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
32752025-07-17T11:33:19.823ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
32762025-07-17T11:33:19.840ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
32772025-07-17T11:33:19.841ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
32782025-07-17T11:33:19.845ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
32792025-07-17T11:33:19.845ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
32802025-07-17T11:33:19.847ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 83.49917ms iteration = 46 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
32812025-07-17T11:33:19.852ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
32822025-07-17T11:33:19.852ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
32832025-07-17T11:33:19.855ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
32842025-07-17T11:33:19.855ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
32852025-07-17T11:33:19.858ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32862025-07-17T11:33:19.862ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 0ae78f24-48db-4cf7-91cb-8d5311f2a17f, lookup_type: ById(0ae78f24-48db-4cf7-91cb-8d5311f2a17f) } result = Ok(())
32872025-07-17T11:33:19.862ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
32882025-07-17T11:33:19.863ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
32892025-07-17T11:33:19.867ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 102.268869ms iteration = 16 status = 0
32902025-07-17T11:33:19.883ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:372
32912025-07-17T11:33:19.883ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 119.316003ms iteration = 16 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
32922025-07-17T11:33:19.890ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 126.538408ms iteration = 17 status = {"physical_disks_added":0}
32932025-07-17T11:33:19.900ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/all-sp-ids
32942025-07-17T11:33:19.900ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/all-sp-ids", status: 200, headers: {"content-type": "application/json", "x-request-id": "f272412e-fb66-456c-877e-cb53535a7517", "content-length": "78", "date": "Thu, 17 Jul 2025 11:33:19 GMT"} })
32952025-07-17T11:33:19.900ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = switch
32962025-07-17T11:33:19.900ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 1 sp_type = sled
32972025-07-17T11:33:19.900ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = sled
32982025-07-17T11:33:19.901ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = switch
32992025-07-17T11:33:19.901ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = sled
33002025-07-17T11:33:19.901ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 1 sp_type = sled
33012025-07-17T11:33:19.903ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 0 sp_type = switch
33022025-07-17T11:33:19.904ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 0 sp_type = sled
33032025-07-17T11:33:19.905ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 1 sp_type = sled
33042025-07-17T11:33:19.908ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 0 sp_type = switch
33052025-07-17T11:33:19.908ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = switch
33062025-07-17T11:33:19.908ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = switch
33072025-07-17T11:33:19.908ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 0 sp_type = sled
33082025-07-17T11:33:19.908ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = sled
33092025-07-17T11:33:19.908ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = sled
33102025-07-17T11:33:19.909ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 1 sp_type = sled
33112025-07-17T11:33:19.909ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 1 sp_type = sled
33122025-07-17T11:33:19.909ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 1 sp_type = sled
33132025-07-17T11:33:19.910ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 0 sp_type = switch start_ena = None
33142025-07-17T11:33:19.910ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
33152025-07-17T11:33:19.911ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "19bdc06a-1c10-4f56-8587-f75baed0707c", "content-length": "81", "date": "Thu, 17 Jul 2025 11:33:19 GMT"} })
33162025-07-17T11:33:19.912ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 1 sp_type = sled start_ena = None
33172025-07-17T11:33:19.912ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/1/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
33182025-07-17T11:33:19.912ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/1/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e91703e-9e82-44c2-89aa-d78debcd22a6", "content-length": "81", "date": "Thu, 17 Jul 2025 11:33:19 GMT"} })
33192025-07-17T11:33:19.913ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
33202025-07-17T11:33:19.913ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
33212025-07-17T11:33:19.913ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
33222025-07-17T11:33:19.913ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
33232025-07-17T11:33:19.933ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 168.794686ms iteration = 16 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
33242025-07-17T11:33:19.970ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 0 sp_type = sled start_ena = None
33252025-07-17T11:33:19.970ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
33262025-07-17T11:33:19.970ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "550b1679-33e0-4138-ab54-35cb074933cd", "content-length": "81", "date": "Thu, 17 Jul 2025 11:33:19 GMT"} })
33272025-07-17T11:33:19.972ZDEBGnexus (BackgroundTasks): activation complete background_task = sp_ereport_ingester elapsed = 208.379707ms iteration = 16 status = {"errors":[],"sps":[]}
33282025-07-17T11:33:19.985ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
33292025-07-17T11:33:19.985ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
33302025-07-17T11:33:19.985ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 221.192356ms iteration = 16 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
33312025-07-17T11:33:20.008ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 275.201688ms iteration = 16 status = {}
33322025-07-17T11:33:20.036ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
33332025-07-17T11:33:20.037ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4419da0-af06-4bdb-9a3c-67e9ca64ada8", "content-length": "691", "date": "Thu, 17 Jul 2025 11:33:20 GMT"} })
33342025-07-17T11:33:20.037ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
33352025-07-17T11:33:20.037ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
33362025-07-17T11:33:20.037ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
33372025-07-17T11:33:20.038ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1194 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 zone_address = fd00:1122:3344:101::2
33382025-07-17T11:33:20.038ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 274.428667ms iteration = 16 status = {}
33392025-07-17T11:33:20.038ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
33402025-07-17T11:33:20.038ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5af7c36-c802-4312-a720-44e4838a7b29", "content-length": "26", "date": "Thu, 17 Jul 2025 11:33:19 GMT"} })
33412025-07-17T11:33:20.038ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1197 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
33422025-07-17T11:33:20.040ZINFOnexus (BackgroundTasks): VPC route manager sled 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
33432025-07-17T11:33:20.040ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
33442025-07-17T11:33:20.040ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
33452025-07-17T11:33:20.040ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b31f0cd-720c-40f0-bd80-7092361fdb83", "content-length": "372", "date": "Thu, 17 Jul 2025 11:33:20 GMT"} })
33462025-07-17T11:33:20.041ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
33472025-07-17T11:33:20.041ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
33482025-07-17T11:33:20.054ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
33492025-07-17T11:33:20.054ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
33502025-07-17T11:33:20.054ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "3a0df18e-b6a6-4e4c-a030-24c5c137222b", "date": "Thu, 17 Jul 2025 11:33:20 GMT"} })
33512025-07-17T11:33:20.054ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
33522025-07-17T11:33:20.055ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
33532025-07-17T11:33:20.057ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
33542025-07-17T11:33:20.057ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
33552025-07-17T11:33:20.061ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
33562025-07-17T11:33:20.061ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
33572025-07-17T11:33:20.063ZINFOnexus (BackgroundTasks): VPC route manager sled 0ae78f24-48db-4cf7-91cb-8d5311f2a17f push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
33582025-07-17T11:33:20.063ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
33592025-07-17T11:33:20.064ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "a5f02c96-8614-476a-af4b-a51ea9236d71", "date": "Thu, 17 Jul 2025 11:33:20 GMT"} })
33602025-07-17T11:33:20.065ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 301.069499ms iteration = 16 status = {}
33612025-07-17T11:33:20.143ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
33622025-07-17T11:33:20.144ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
33632025-07-17T11:33:20.146ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 151caa7c-6a05-4995-94b4-7946d428117d, rack_id: 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5, switch_location: "switch0", port_name: Name(Name("qsfp0")), port_settings_id: Some(2e30c7ae-26da-46e4-9127-aac194921f20) } rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0
33642025-07-17T11:33:20.146ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
33652025-07-17T11:33:20.146ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
33662025-07-17T11:33:20.163ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1832 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = Name(Name("qsfp0"))
33672025-07-17T11:33:20.163ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1859 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0 switch_port_id = Name(Name("qsfp0")) switch_port_settings_id = 2e30c7ae-26da-46e4-9127-aac194921f20
33682025-07-17T11:33:20.163ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
33692025-07-17T11:33:20.163ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
33702025-07-17T11:33:20.163ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
33712025-07-17T11:33:20.163ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
33722025-07-17T11:33:20.163ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
33732025-07-17T11:33:20.163ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2006 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
33742025-07-17T11:33:20.163ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
33752025-07-17T11:33:20.163ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2042 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
33762025-07-17T11:33:20.163ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
33772025-07-17T11:33:20.163ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
33782025-07-17T11:33:20.164ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
33792025-07-17T11:33:20.166ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0
33802025-07-17T11:33:20.166ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
33812025-07-17T11:33:20.401ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "d648d87b-1dfa-4b99-b423-05bb456cb6b0", "date": "Thu, 17 Jul 2025 11:33:20 GMT"} })
33822025-07-17T11:33:20.401ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch = Switch0
33832025-07-17T11:33:20.401ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
33842025-07-17T11:33:20.401ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "6043fd31-0856-4b1e-b3a6-81415b14b530", "content-length": "682", "date": "Thu, 17 Jul 2025 11:33:20 GMT"} })
33852025-07-17T11:33:20.401ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
33862025-07-17T11:33:20.402ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
33872025-07-17T11:33:20.405ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
33882025-07-17T11:33:20.405ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
33892025-07-17T11:33:20.409ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
33902025-07-17T11:33:20.409ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
33912025-07-17T11:33:20.411ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
33922025-07-17T11:33:20.411ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
33932025-07-17T11:33:20.417ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 653.180851ms iteration = 16 status = {}
33942025-07-17T11:33:21.692ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 344 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = 0ccdb862-b441-480b-bdaa-492bf9e90970 response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
33952025-07-17T11:33:24.764ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 92 reason = Timeout
33962025-07-17T11:33:24.764ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
33972025-07-17T11:33:24.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
33982025-07-17T11:33:24.765ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33992025-07-17T11:33:24.768ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
34002025-07-17T11:33:24.768ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
34012025-07-17T11:33:24.768ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
34022025-07-17T11:33:24.770ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
34032025-07-17T11:33:24.770ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 6.620237ms iteration = 92 status = {"chicken_switches_updated":false}
34042025-07-17T11:33:29.764ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 47 reason = Timeout
34052025-07-17T11:33:29.764ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 47 reason = Timeout
34062025-07-17T11:33:29.764ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
34072025-07-17T11:33:29.764ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 93 reason = Timeout
34082025-07-17T11:33:29.764ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
34092025-07-17T11:33:29.764ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
34102025-07-17T11:33:29.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
34112025-07-17T11:33:29.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
34122025-07-17T11:33:29.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
34132025-07-17T11:33:29.766ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34142025-07-17T11:33:29.767ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34152025-07-17T11:33:29.768ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
34162025-07-17T11:33:29.768ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
34172025-07-17T11:33:29.769ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 5.25318ms iteration = 47 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
34182025-07-17T11:33:29.770ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
34192025-07-17T11:33:29.770ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
34202025-07-17T11:33:29.770ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
34212025-07-17T11:33:29.770ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
34222025-07-17T11:33:29.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
34232025-07-17T11:33:29.771ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
34242025-07-17T11:33:29.772ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
34252025-07-17T11:33:29.772ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 7.718472ms iteration = 93 status = {"chicken_switches_updated":false}
34262025-07-17T11:33:29.775ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
34272025-07-17T11:33:29.775ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
34282025-07-17T11:33:29.776ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34292025-07-17T11:33:29.779ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
34302025-07-17T11:33:29.779ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
34312025-07-17T11:33:29.779ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
34322025-07-17T11:33:29.793ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
34332025-07-17T11:33:29.793ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 28.654628ms iteration = 47 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
34342025-07-17T11:33:31.684ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:35131 req_id = a419f792-925b-48d9-8fa4-db9e7bc97789 roles = RoleSet { roles: {} } uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?sort_by=id_ascending
34352025-07-17T11:33:31.684ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:35131 req_id = a419f792-925b-48d9-8fa4-db9e7bc97789 resource = Database result = Ok(()) uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?sort_by=id_ascending
34362025-07-17T11:33:31.687ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 3000 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:35131 req_id = a419f792-925b-48d9-8fa4-db9e7bc97789 response_code = 200 uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?sort_by=id_ascending
34372025-07-17T11:33:31.687ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:35131 req_id = 47f2a2d8-33c6-425b-a3fb-35d05c14a544 roles = RoleSet { roles: {} } uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmMzlkM2U3MC02NmU0LTQ2OGEtYTQ0ZC03NDZkMjJiZWEzYTAifX0%3D
34382025-07-17T11:33:31.687ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:35131 req_id = 47f2a2d8-33c6-425b-a3fb-35d05c14a544 resource = Database result = Ok(()) uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmMzlkM2U3MC02NmU0LTQ2OGEtYTQ0ZC03NDZkMjJiZWEzYTAifX0%3D
34392025-07-17T11:33:31.689ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 1860 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:35131 req_id = 47f2a2d8-33c6-425b-a3fb-35d05c14a544 response_code = 200 uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmMzlkM2U3MC02NmU0LTQ2OGEtYTQ0ZC03NDZkMjJiZWEzYTAifX0%3D
34402025-07-17T11:33:31.692ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 336 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = 6b8713d6-654e-44c7-a0cf-1f10bc3033dc response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
34412025-07-17T11:33:34.764ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 94 reason = Timeout
34422025-07-17T11:33:34.764ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
34432025-07-17T11:33:34.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
34442025-07-17T11:33:34.766ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34452025-07-17T11:33:34.768ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
34462025-07-17T11:33:34.768ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
34472025-07-17T11:33:34.768ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
34482025-07-17T11:33:34.770ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
34492025-07-17T11:33:34.770ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 6.542916ms iteration = 94 status = {"chicken_switches_updated":false}
34502025-07-17T11:33:39.764ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 48 reason = Timeout
34512025-07-17T11:33:39.764ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
34522025-07-17T11:33:39.764ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 48 reason = Timeout
34532025-07-17T11:33:39.764ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
34542025-07-17T11:33:39.764ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 95 reason = Timeout
34552025-07-17T11:33:39.764ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
34562025-07-17T11:33:39.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
34572025-07-17T11:33:39.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
34582025-07-17T11:33:39.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
34592025-07-17T11:33:39.766ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34602025-07-17T11:33:39.766ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
34612025-07-17T11:33:39.767ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
34622025-07-17T11:33:39.767ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34632025-07-17T11:33:39.768ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.5847ms iteration = 48 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
34642025-07-17T11:33:39.769ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
34652025-07-17T11:33:39.769ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
34662025-07-17T11:33:39.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
34672025-07-17T11:33:39.770ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
34682025-07-17T11:33:39.770ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
34692025-07-17T11:33:39.771ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
34702025-07-17T11:33:39.772ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
34712025-07-17T11:33:39.773ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
34722025-07-17T11:33:39.773ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
34732025-07-17T11:33:39.773ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 9.170771ms iteration = 95 status = {"chicken_switches_updated":false}
34742025-07-17T11:33:39.774ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34752025-07-17T11:33:39.776ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
34762025-07-17T11:33:39.776ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
34772025-07-17T11:33:39.776ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
34782025-07-17T11:33:39.787ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
34792025-07-17T11:33:39.787ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 23.044844ms iteration = 48 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
34802025-07-17T11:33:41.693ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 482 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = 3043630e-25eb-485d-8286-b28ca78bd18f response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
34812025-07-17T11:33:44.763ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 96 reason = Timeout
34822025-07-17T11:33:44.764ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
34832025-07-17T11:33:44.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
34842025-07-17T11:33:44.765ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34852025-07-17T11:33:44.768ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
34862025-07-17T11:33:44.768ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
34872025-07-17T11:33:44.768ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
34882025-07-17T11:33:44.770ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
34892025-07-17T11:33:44.770ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 6.993533ms iteration = 96 status = {"chicken_switches_updated":false}
34902025-07-17T11:33:49.676ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 9 reason = Timeout
34912025-07-17T11:33:49.676ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 9 reason = Timeout
34922025-07-17T11:33:49.676ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 11 reason = Timeout
34932025-07-17T11:33:49.676ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
34942025-07-17T11:33:49.676ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 11 reason = Timeout
34952025-07-17T11:33:49.676ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 17 reason = Timeout
34962025-07-17T11:33:49.676ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
34972025-07-17T11:33:49.676ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 9 reason = Timeout
34982025-07-17T11:33:49.676ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-07-17 11:23:49.676542216 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
34992025-07-17T11:33:49.676ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
35002025-07-17T11:33:49.676ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 9 reason = Timeout
35012025-07-17T11:33:49.676ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
35022025-07-17T11:33:49.676ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 9 reason = Timeout
35032025-07-17T11:33:49.676ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
35042025-07-17T11:33:49.676ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 9 reason = Timeout
35052025-07-17T11:33:49.676ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
35062025-07-17T11:33:49.676ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
35072025-07-17T11:33:49.677ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("3dd3bf3b-7760-4750-b1d3-52ee777335f0.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("79e5c77d-18b3-4951-8d11-cfa3eae2b3e3.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("79e5c77d-18b3-4951-8d11-cfa3eae2b3e3.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::8)) }], valid_until: Instant { tv_sec: 3823, tv_nsec: 429301857 } })
35082025-07-17T11:33:49.677ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
35092025-07-17T11:33:49.677ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
35102025-07-17T11:33:49.677ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
35112025-07-17T11:33:49.677ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("3144a75a-317f-4b68-93e2-a210ca1189d1.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("87408888-9df4-4e0c-bc8e-2f920d039100.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("ee4378c5-e5f2-41cf-8969-e5e03e1b9a41.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("ee4378c5-e5f2-41cf-8969-e5e03e1b9a41.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:3::1)) }], valid_until: Instant { tv_sec: 3823, tv_nsec: 429501500 } })
35122025-07-17T11:33:49.677ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
35132025-07-17T11:33:49.677ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
35142025-07-17T11:33:49.677ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
35152025-07-17T11:33:49.677ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.139295ms iteration = 9 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
35162025-07-17T11:33:49.677ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } dns_group = internal
35172025-07-17T11:33:49.677ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.201246ms iteration = 9 status = {"addresses":["[fd00:1122:3344:1::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:3::1]:5353"]}
35182025-07-17T11:33:49.678ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
35192025-07-17T11:33:49.678ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
35202025-07-17T11:33:49.679ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35212025-07-17T11:33:49.679ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
35222025-07-17T11:33:49.679ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 10 reason = Timeout
35232025-07-17T11:33:49.679ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:85 reason = blueprint disabled target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a
35242025-07-17T11:33:49.679ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 10.99\u{b5}s iteration = 10 status = {"enabled":false,"target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a"}
35252025-07-17T11:33:49.679ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
35262025-07-17T11:33:49.680ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3823, tv_nsec: 432467379 } })
35272025-07-17T11:33:49.680ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1186
35282025-07-17T11:33:49.681ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35292025-07-17T11:33:49.681ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-07-17 11:23:49.676542216 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
35302025-07-17T11:33:49.681ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
35312025-07-17T11:33:49.681ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 4.940155ms iteration = 9 status = {"expiration":"2025-07-17T11:23:49.676542216Z","pruned":[]}
35322025-07-17T11:33:49.681ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
35332025-07-17T11:33:49.681ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
35342025-07-17T11:33:49.682ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35352025-07-17T11:33:49.684ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
35362025-07-17T11:33:49.684ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
35372025-07-17T11:33:49.685ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_planner iteration = 11 reason = Timeout
35382025-07-17T11:33:49.685ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 9 reason = Timeout
35392025-07-17T11:33:49.685ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:53
35402025-07-17T11:33:49.685ZDEBGnexus (BackgroundTasks): blueprint planning disabled, doing nothing background_task = blueprint_planner
35412025-07-17T11:33:49.685ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 9 reason = Timeout
35422025-07-17T11:33:49.685ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
35432025-07-17T11:33:49.685ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_planner elapsed = 17.09\u{b5}s iteration = 11 status = "Disabled"
35442025-07-17T11:33:49.685ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
35452025-07-17T11:33:49.685ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
35462025-07-17T11:33:49.685ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
35472025-07-17T11:33:49.686ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35482025-07-17T11:33:49.686ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 9 reason = Timeout
35492025-07-17T11:33:49.686ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
35502025-07-17T11:33:49.686ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
35512025-07-17T11:33:49.686ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
35522025-07-17T11:33:49.687ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
35532025-07-17T11:33:49.687ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35542025-07-17T11:33:49.688ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
35552025-07-17T11:33:49.688ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
35562025-07-17T11:33:49.688ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
35572025-07-17T11:33:49.688ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
35582025-07-17T11:33:49.688ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
35592025-07-17T11:33:49.689ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
35602025-07-17T11:33:49.689ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
35612025-07-17T11:33:49.689ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
35622025-07-17T11:33:49.690ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35632025-07-17T11:33:49.691ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
35642025-07-17T11:33:49.691ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
35652025-07-17T11:33:49.691ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
35662025-07-17T11:33:49.692ZDEBGnexus (BackgroundTasks): activating background_task = webhook_deliverator iteration = 9 reason = Timeout
35672025-07-17T11:33:49.692ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 9 reason = Timeout
35682025-07-17T11:33:49.692ZDEBGnexus (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} }
35692025-07-17T11:33:49.692ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
35702025-07-17T11:33:49.692ZDEBGnexus (BackgroundTasks): activating background_task = alert_dispatcher iteration = 9 reason = Timeout
35712025-07-17T11:33:49.692ZDEBGnexus (BackgroundTasks): roles background_task = alert_dispatcher roles = RoleSet { roles: {} }
35722025-07-17T11:33:49.692ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = alert_dispatcher resource = Database result = Ok(())
35732025-07-17T11:33:49.692ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
35742025-07-17T11:33:49.692ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database result = Ok(())
35752025-07-17T11:33:49.694ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
35762025-07-17T11:33:49.694ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
35772025-07-17T11:33:49.694ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
35782025-07-17T11:33:49.696ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 3.855641ms iteration = 9 status = {"errors":[],"found_port_ok":[]}
35792025-07-17T11:33:49.709ZDEBGnexus (BackgroundTasks): roles background_task = alert_dispatcher roles = RoleSet { roles: {} }
35802025-07-17T11:33:49.709ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = alert_dispatcher resource = Database result = Ok(())
35812025-07-17T11:33:49.710ZINFOnexus (BackgroundTasks): all webhook delivery tasks completed background_task = webhook_deliverator file = nexus/src/app/background/tasks/webhook_deliverator.rs:204 num_receivers = 0
35822025-07-17T11:33:49.710ZDEBGnexus (BackgroundTasks): activation complete background_task = webhook_deliverator elapsed = 18.306301ms iteration = 9 status = {"by_rx":{},"error":null}
35832025-07-17T11:33:49.713ZDEBGnexus (BackgroundTasks): activation complete background_task = alert_dispatcher elapsed = 21.407533ms iteration = 9 status = {"dispatched":[],"errors":[],"glob_version":"161.0.0","globs_reprocessed":{},"no_receivers":[]}
35842025-07-17T11:33:49.733ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 17 reason = Timeout
35852025-07-17T11:33:49.733ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 17 reason = Timeout
35862025-07-17T11:33:49.733ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
35872025-07-17T11:33:49.733ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
35882025-07-17T11:33:49.736ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3823, tv_nsec: 489231508 } })
35892025-07-17T11:33:49.737ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1186
35902025-07-17T11:33:49.738ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 5.585704ms iteration = 17 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
35912025-07-17T11:33:49.747ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
35922025-07-17T11:33:49.747ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:170 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
35932025-07-17T11:33:49.747ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 61.309959ms iteration = 9 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
35942025-07-17T11:33:49.747ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
35952025-07-17T11:33:49.748ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
35962025-07-17T11:33:49.748ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
35972025-07-17T11:33:49.748ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
35982025-07-17T11:33:49.748ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
35992025-07-17T11:33:49.748ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
36002025-07-17T11:33:49.748ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
36012025-07-17T11:33:49.749ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36022025-07-17T11:33:49.750ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36032025-07-17T11:33:49.752ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
36042025-07-17T11:33:49.752ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 915fbba0-947f-4bf0-9f9f-48a79df81f7b dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
36052025-07-17T11:33:49.752ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
36062025-07-17T11:33:49.752ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
36072025-07-17T11:33:49.753ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
36082025-07-17T11:33:49.753ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
36092025-07-17T11:33:49.753ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
36102025-07-17T11:33:49.754ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36112025-07-17T11:33:49.755ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
36122025-07-17T11:33:49.755ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:138 found = 0 not_ready_to_be_deleted = 0
36132025-07-17T11:33:49.755ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 70.249986ms iteration = 9 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
36142025-07-17T11:33:49.756ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
36152025-07-17T11:33:49.756ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
36162025-07-17T11:33:49.756ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
36172025-07-17T11:33:49.758ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
36182025-07-17T11:33:49.758ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 4a9e4c5e-728e-4850-bd58-14548c02bf6f dns_zone_name = oxide.test found_so_far = 0 version = 2
36192025-07-17T11:33:49.758ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
36202025-07-17T11:33:49.758ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
36212025-07-17T11:33:49.759ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
36222025-07-17T11:33:49.759ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
36232025-07-17T11:33:49.759ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
36242025-07-17T11:33:49.760ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36252025-07-17T11:33:49.762ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 915fbba0-947f-4bf0-9f9f-48a79df81f7b dns_zone_name = control-plane.oxide.internal found_so_far = 59 version = 1
36262025-07-17T11:33:49.762ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
36272025-07-17T11:33:49.762ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-07-17 11:26:34.922237 UTC dns_group = internal generation = 1
36282025-07-17T11:33:49.762ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 85.514097ms iteration = 9 status = {"generation":1}
36292025-07-17T11:33:49.764ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 17 reason = Timeout
36302025-07-17T11:33:49.764ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 17 reason = Timeout
36312025-07-17T11:33:49.764ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
36322025-07-17T11:33:49.764ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
36332025-07-17T11:33:49.764ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 17 reason = Timeout
36342025-07-17T11:33:49.764ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
36352025-07-17T11:33:49.764ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
36362025-07-17T11:33:49.764ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 49 reason = Timeout
36372025-07-17T11:33:49.764ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
36382025-07-17T11:33:49.764ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 17 reason = Timeout
36392025-07-17T11:33:49.764ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 49 reason = Timeout
36402025-07-17T11:33:49.764ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
36412025-07-17T11:33:49.764ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
36422025-07-17T11:33:49.764ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 97 reason = Timeout
36432025-07-17T11:33:49.764ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
36442025-07-17T11:33:49.764ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 17 reason = Timeout
36452025-07-17T11:33:49.764ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
36462025-07-17T11:33:49.764ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 17 reason = Timeout
36472025-07-17T11:33:49.764ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
36482025-07-17T11:33:49.764ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 17 reason = Timeout
36492025-07-17T11:33:49.764ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
36502025-07-17T11:33:49.764ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 17 reason = Timeout
36512025-07-17T11:33:49.764ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
36522025-07-17T11:33:49.764ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 17 reason = Timeout
36532025-07-17T11:33:49.764ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
36542025-07-17T11:33:49.764ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 18 reason = Timeout
36552025-07-17T11:33:49.764ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
36562025-07-17T11:33:49.764ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 17 reason = Timeout
36572025-07-17T11:33:49.764ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
36582025-07-17T11:33:49.764ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 17 reason = Timeout
36592025-07-17T11:33:49.764ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
36602025-07-17T11:33:49.764ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 17 reason = Timeout
36612025-07-17T11:33:49.764ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
36622025-07-17T11:33:49.764ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
36632025-07-17T11:33:49.764ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 17 reason = Timeout
36642025-07-17T11:33:49.764ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
36652025-07-17T11:33:49.764ZDEBGnexus (BackgroundTasks): activating background_task = sp_ereport_ingester iteration = 17 reason = Timeout
36662025-07-17T11:33:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
36672025-07-17T11:33:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
36682025-07-17T11:33:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
36692025-07-17T11:33:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
36702025-07-17T11:33:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
36712025-07-17T11:33:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
36722025-07-17T11:33:49.765ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
36732025-07-17T11:33:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
36742025-07-17T11:33:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
36752025-07-17T11:33:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
36762025-07-17T11:33:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
36772025-07-17T11:33:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
36782025-07-17T11:33:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
36792025-07-17T11:33:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
36802025-07-17T11:33:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
36812025-07-17T11:33:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
36822025-07-17T11:33:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
36832025-07-17T11:33:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
36842025-07-17T11:33:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
36852025-07-17T11:33:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
36862025-07-17T11:33:49.765ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12225, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3823, tv_nsec: 517768514 } })
36872025-07-17T11:33:49.766ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36882025-07-17T11:33:49.769ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 4.699382ms iteration = 17 status = {"errors":[],"garbage_collect_requested":[]}
36892025-07-17T11:33:49.769ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
36902025-07-17T11:33:49.769ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
36912025-07-17T11:33:49.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
36922025-07-17T11:33:49.770ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36932025-07-17T11:33:49.774ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36942025-07-17T11:33:49.774ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
36952025-07-17T11:33:49.774ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
36962025-07-17T11:33:49.774ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
36972025-07-17T11:33:49.775ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36982025-07-17T11:33:49.776ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
36992025-07-17T11:33:49.776ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
37002025-07-17T11:33:49.777ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37012025-07-17T11:33:49.778ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
37022025-07-17T11:33:49.778ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
37032025-07-17T11:33:49.778ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
37042025-07-17T11:33:49.778ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37052025-07-17T11:33:49.778ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
37062025-07-17T11:33:49.778ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
37072025-07-17T11:33:49.778ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
37082025-07-17T11:33:49.778ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
37092025-07-17T11:33:49.778ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
37102025-07-17T11:33:49.778ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
37112025-07-17T11:33:49.779ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 4a9e4c5e-728e-4850-bd58-14548c02bf6f dns_zone_name = oxide.test found_so_far = 4 version = 2
37122025-07-17T11:33:49.780ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
37132025-07-17T11:33:49.780ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-07-17 11:26:35.847594 UTC dns_group = external generation = 2
37142025-07-17T11:33:49.780ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 103.380083ms iteration = 9 status = {"generation":2}
37152025-07-17T11:33:49.780ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
37162025-07-17T11:33:49.780ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37172025-07-17T11:33:49.780ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
37182025-07-17T11:33:49.781ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
37192025-07-17T11:33:49.781ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
37202025-07-17T11:33:49.781ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
37212025-07-17T11:33:49.781ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
37222025-07-17T11:33:49.781ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
37232025-07-17T11:33:49.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
37242025-07-17T11:33:49.782ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 17.561312ms iteration = 17 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
37252025-07-17T11:33:49.782ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
37262025-07-17T11:33:49.783ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
37272025-07-17T11:33:49.784ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
37282025-07-17T11:33:49.784ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
37292025-07-17T11:33:49.784ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
37302025-07-17T11:33:49.784ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3823, tv_nsec: 536820255 } })
37312025-07-17T11:33:49.785ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1186 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
37322025-07-17T11:33:49.792ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:550 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
37332025-07-17T11:33:49.792ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 27.506842ms iteration = 17 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
37342025-07-17T11:33:49.845ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-07-17T11:25:46.964149Z, time_modified: 2025-07-17T11:25:46.964149Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 1fa836d9-7854-4ada-8c89-c26d922646c6, db_silo: Silo { identity: SiloIdentity { id: 1fa836d9-7854-4ada-8c89-c26d922646c6, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-07-17T11:26:35.643175Z, time_modified: 2025-07-17T11:26:35.643175Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "d7c3455b70a66c86ec2a7e597ca46fe0ea0c49dde5c1363f198c20aa77d58442" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 1fa836d9-7854-4ada-8c89-c26d922646c6, db_silo: Silo { identity: SiloIdentity { id: 1fa836d9-7854-4ada-8c89-c26d922646c6, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-07-17T11:26:35.643175Z, time_modified: 2025-07-17T11:26:35.643175Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "d7c3455b70a66c86ec2a7e597ca46fe0ea0c49dde5c1363f198c20aa77d58442" }] }) }
37352025-07-17T11:33:49.845ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 169.351223ms iteration = 9 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"1fa836d9-7854-4ada-8c89-c26d922646c6","tls_certs":["d7c3455b70a66c86ec2a7e597ca46fe0ea0c49dde5c1363f198c20aa77d58442"]}},"default_endpoint":{"silo_id":"1fa836d9-7854-4ada-8c89-c26d922646c6","tls_certs":["d7c3455b70a66c86ec2a7e597ca46fe0ea0c49dde5c1363f198c20aa77d58442"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
37362025-07-17T11:33:49.847ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37372025-07-17T11:33:49.853ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
37382025-07-17T11:33:49.853ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
37392025-07-17T11:33:49.853ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
37402025-07-17T11:33:49.857ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37412025-07-17T11:33:49.859ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
37422025-07-17T11:33:49.859ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
37432025-07-17T11:33:49.859ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
37442025-07-17T11:33:49.859ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 94.862221ms iteration = 97 status = {"chicken_switches_updated":false}
37452025-07-17T11:33:49.859ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
37462025-07-17T11:33:49.860ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
37472025-07-17T11:33:49.860ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
37482025-07-17T11:33:49.860ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
37492025-07-17T11:33:49.860ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 95.997015ms iteration = 17 status = {"errors":[],"requests_created_ok":[]}
37502025-07-17T11:33:49.861ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
37512025-07-17T11:33:49.861ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
37522025-07-17T11:33:49.862ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
37532025-07-17T11:33:49.870ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37542025-07-17T11:33:49.871ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37552025-07-17T11:33:49.874ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
37562025-07-17T11:33:49.874ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
37572025-07-17T11:33:49.874ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
37582025-07-17T11:33:49.874ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
37592025-07-17T11:33:49.874ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
37602025-07-17T11:33:49.874ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
37612025-07-17T11:33:49.875ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
37622025-07-17T11:33:49.876ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
37632025-07-17T11:33:49.876ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
37642025-07-17T11:33:49.876ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
37652025-07-17T11:33:49.882ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 117.236775ms iteration = 17 status = {"errors":[],"finish_invoked_ok":[]}
37662025-07-17T11:33:49.883ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
37672025-07-17T11:33:49.884ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
37682025-07-17T11:33:49.886ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37692025-07-17T11:33:49.890ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
37702025-07-17T11:33:49.890ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
37712025-07-17T11:33:49.890ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
37722025-07-17T11:33:49.897ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 132.581299ms iteration = 18 status = {"physical_disks_added":0}
37732025-07-17T11:33:49.900ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
37742025-07-17T11:33:49.917ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 152.792975ms iteration = 49 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
37752025-07-17T11:33:49.922ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
37762025-07-17T11:33:49.923ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 158.345378ms iteration = 49 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
37772025-07-17T11:33:49.930ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
37782025-07-17T11:33:49.930ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
37792025-07-17T11:33:49.933ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
37802025-07-17T11:33:49.933ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
37812025-07-17T11:33:49.949ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
37822025-07-17T11:33:49.949ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
37832025-07-17T11:33:49.951ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
37842025-07-17T11:33:49.952ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
37852025-07-17T11:33:49.964ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 199.679172ms iteration = 17 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
37862025-07-17T11:33:49.965ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:372
37872025-07-17T11:33:49.965ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 200.582775ms iteration = 17 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
37882025-07-17T11:33:49.973ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
37892025-07-17T11:33:49.973ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
37902025-07-17T11:33:49.973ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
37912025-07-17T11:33:49.974ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
37922025-07-17T11:33:49.978ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1194 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 zone_address = fd00:1122:3344:101::2
37932025-07-17T11:33:49.978ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
37942025-07-17T11:33:49.978ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b8c2c81-bee5-4726-a08a-063f9159a407", "content-length": "26", "date": "Thu, 17 Jul 2025 11:33:49 GMT"} })
37952025-07-17T11:33:49.978ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1197 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
37962025-07-17T11:33:49.981ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
37972025-07-17T11:33:50.003ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
37982025-07-17T11:33:50.003ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
37992025-07-17T11:33:50.014ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38002025-07-17T11:33:50.017ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 253.31799ms iteration = 17 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
38012025-07-17T11:33:50.019ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 0ae78f24-48db-4cf7-91cb-8d5311f2a17f, lookup_type: ById(0ae78f24-48db-4cf7-91cb-8d5311f2a17f) } result = Ok(())
38022025-07-17T11:33:50.019ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
38032025-07-17T11:33:50.019ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
38042025-07-17T11:33:50.021ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 257.09872ms iteration = 17 status = 0
38052025-07-17T11:33:50.022ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
38062025-07-17T11:33:50.022ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
38072025-07-17T11:33:50.022ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 257.501275ms iteration = 17 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
38082025-07-17T11:33:50.029ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1194 zone_address = fd00:1122:3344:101::2
38092025-07-17T11:33:50.029ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
38102025-07-17T11:33:50.029ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "8112607c-d410-43e7-8733-aaf83e93cfe3", "content-length": "26", "date": "Thu, 17 Jul 2025 11:33:49 GMT"} })
38112025-07-17T11:33:50.030ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1197 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
38122025-07-17T11:33:50.031ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
38132025-07-17T11:33:50.048ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
38142025-07-17T11:33:50.049ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1194 zone_address = fd00:1122:3344:101::2
38152025-07-17T11:33:50.049ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
38162025-07-17T11:33:50.050ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "12f6f9de-0db5-4710-90a1-24459dc0a5d9", "content-length": "26", "date": "Thu, 17 Jul 2025 11:33:49 GMT"} })
38172025-07-17T11:33:50.050ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1197 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
38182025-07-17T11:33:50.051ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
38192025-07-17T11:33:50.064ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/all-sp-ids
38202025-07-17T11:33:50.065ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/all-sp-ids", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8072ba4-e5e8-497c-b4a9-184d2b66c2a6", "content-length": "78", "date": "Thu, 17 Jul 2025 11:33:49 GMT"} })
38212025-07-17T11:33:50.065ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 1 sp_type = sled
38222025-07-17T11:33:50.065ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = switch
38232025-07-17T11:33:50.065ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = sled
38242025-07-17T11:33:50.065ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = switch
38252025-07-17T11:33:50.065ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = sled
38262025-07-17T11:33:50.065ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 1 sp_type = sled
38272025-07-17T11:33:50.068ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 1 sp_type = sled
38282025-07-17T11:33:50.070ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 1 sp_type = sled
38292025-07-17T11:33:50.070ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 1 sp_type = sled
38302025-07-17T11:33:50.070ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 1 sp_type = sled
38312025-07-17T11:33:50.103ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 0 sp_type = switch
38322025-07-17T11:33:50.104ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 0 sp_type = sled
38332025-07-17T11:33:50.106ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 0 sp_type = switch
38342025-07-17T11:33:50.106ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = switch
38352025-07-17T11:33:50.106ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = switch
38362025-07-17T11:33:50.107ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 0 sp_type = sled
38372025-07-17T11:33:50.107ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = sled
38382025-07-17T11:33:50.107ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = sled
38392025-07-17T11:33:50.126ZINFOnexus (BackgroundTasks): VPC route manager sled 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
38402025-07-17T11:33:50.126ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
38412025-07-17T11:33:50.126ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 1 sp_type = sled start_ena = None
38422025-07-17T11:33:50.126ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/1/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
38432025-07-17T11:33:50.126ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "89c854ae-d32f-4bdc-a99c-e5da1665759f", "content-length": "372", "date": "Thu, 17 Jul 2025 11:33:50 GMT"} })
38442025-07-17T11:33:50.126ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
38452025-07-17T11:33:50.127ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/1/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "9adc5346-74fd-4c6c-92a9-72d87b70efd0", "content-length": "81", "date": "Thu, 17 Jul 2025 11:33:49 GMT"} })
38462025-07-17T11:33:50.127ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
38472025-07-17T11:33:50.154ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
38482025-07-17T11:33:50.154ZINFOnexus (BackgroundTasks): instance reincarnation completed successfully background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:113 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0
38492025-07-17T11:33:50.154ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 469.385718ms iteration = 9 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
38502025-07-17T11:33:50.154ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "c3306aff-8186-4b08-9636-59669836e6f6", "date": "Thu, 17 Jul 2025 11:33:50 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
38512025-07-17T11:33:50.154ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
38522025-07-17T11:33:50.154ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 0 sp_type = switch start_ena = None
38532025-07-17T11:33:50.154ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
38542025-07-17T11:33:50.155ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7df7a0f-d5b5-45e4-a312-b9bfc2713e15", "content-length": "81", "date": "Thu, 17 Jul 2025 11:33:49 GMT"} })
38552025-07-17T11:33:50.156ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "d19030b9-f93f-470a-b428-da4aaf13cfd3", "date": "Thu, 17 Jul 2025 11:33:50 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
38562025-07-17T11:33:50.156ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
38572025-07-17T11:33:50.159ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
38582025-07-17T11:33:50.159ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 482.610403ms iteration = 11 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
38592025-07-17T11:33:50.160ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 0 sp_type = sled start_ena = None
38602025-07-17T11:33:50.160ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
38612025-07-17T11:33:50.160ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "46a0a5f0-c147-494d-aa84-bece50717c9b", "content-length": "81", "date": "Thu, 17 Jul 2025 11:33:49 GMT"} })
38622025-07-17T11:33:50.162ZDEBGnexus (BackgroundTasks): activation complete background_task = sp_ereport_ingester elapsed = 397.959466ms iteration = 17 status = {"errors":[],"sps":[]}
38632025-07-17T11:33:50.175ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
38642025-07-17T11:33:50.176ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
38652025-07-17T11:33:50.178ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 501.753975ms iteration = 17 status = null
38662025-07-17T11:33:50.178ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
38672025-07-17T11:33:50.178ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
38682025-07-17T11:33:50.179ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38692025-07-17T11:33:50.181ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
38702025-07-17T11:33:50.181ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
38712025-07-17T11:33:50.182ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
38722025-07-17T11:33:50.182ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
38732025-07-17T11:33:50.182ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "57973081-3a5a-4474-887e-94be414c7183", "date": "Thu, 17 Jul 2025 11:33:50 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
38742025-07-17T11:33:50.182ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
38752025-07-17T11:33:50.184ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "a29df579-1b1d-44be-9c2c-67affdd91819", "date": "Thu, 17 Jul 2025 11:33:50 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
38762025-07-17T11:33:50.184ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
38772025-07-17T11:33:50.185ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "f6844c68-511a-402f-9f08-63b84d3dee45", "date": "Thu, 17 Jul 2025 11:33:50 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
38782025-07-17T11:33:50.185ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
38792025-07-17T11:33:50.187ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
38802025-07-17T11:33:50.187ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 510.859095ms iteration = 11 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
38812025-07-17T11:33:50.191ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
38822025-07-17T11:33:50.191ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
38832025-07-17T11:33:50.191ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "5192c0dd-8190-4641-bd2b-0ced3655a093", "date": "Thu, 17 Jul 2025 11:33:50 GMT"} })
38842025-07-17T11:33:50.191ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
38852025-07-17T11:33:50.192ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
38862025-07-17T11:33:50.194ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
38872025-07-17T11:33:50.194ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
38882025-07-17T11:33:50.194ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 461.028958ms iteration = 17 status = {}
38892025-07-17T11:33:50.197ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
38902025-07-17T11:33:50.197ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
38912025-07-17T11:33:50.200ZINFOnexus (BackgroundTasks): VPC route manager sled 0ae78f24-48db-4cf7-91cb-8d5311f2a17f push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
38922025-07-17T11:33:50.200ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
38932025-07-17T11:33:50.200ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "b12d97a9-03c1-4777-ae18-fc51c1cd72d9", "date": "Thu, 17 Jul 2025 11:33:50 GMT"} })
38942025-07-17T11:33:50.203ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 438.806485ms iteration = 17 status = {}
38952025-07-17T11:33:50.224ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38962025-07-17T11:33:50.226ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
38972025-07-17T11:33:50.226ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
38982025-07-17T11:33:50.226ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
38992025-07-17T11:33:50.254ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
39002025-07-17T11:33:50.254ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
39012025-07-17T11:33:50.256ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 151caa7c-6a05-4995-94b4-7946d428117d, rack_id: 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5, switch_location: "switch0", port_name: Name(Name("qsfp0")), port_settings_id: Some(2e30c7ae-26da-46e4-9127-aac194921f20) } rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0
39022025-07-17T11:33:50.256ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
39032025-07-17T11:33:50.256ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
39042025-07-17T11:33:50.269ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1832 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = Name(Name("qsfp0"))
39052025-07-17T11:33:50.269ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1859 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0 switch_port_id = Name(Name("qsfp0")) switch_port_settings_id = 2e30c7ae-26da-46e4-9127-aac194921f20
39062025-07-17T11:33:50.270ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
39072025-07-17T11:33:50.270ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
39082025-07-17T11:33:50.270ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
39092025-07-17T11:33:50.270ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
39102025-07-17T11:33:50.270ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
39112025-07-17T11:33:50.270ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2006 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
39122025-07-17T11:33:50.270ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
39132025-07-17T11:33:50.270ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2042 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
39142025-07-17T11:33:50.270ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
39152025-07-17T11:33:50.270ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
39162025-07-17T11:33:50.270ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
39172025-07-17T11:33:50.272ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
39182025-07-17T11:33:50.273ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0
39192025-07-17T11:33:50.273ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
39202025-07-17T11:33:50.273ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b0f3295-a767-4b7a-b0f5-e339a82647ed", "content-length": "691", "date": "Thu, 17 Jul 2025 11:33:50 GMT"} })
39212025-07-17T11:33:50.273ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
39222025-07-17T11:33:50.273ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
39232025-07-17T11:33:50.273ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
39242025-07-17T11:33:50.274ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 509.750971ms iteration = 17 status = {}
39252025-07-17T11:33:50.382ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "a9bc54ab-c747-4d11-aedb-3c48578296f7", "date": "Thu, 17 Jul 2025 11:33:50 GMT"} })
39262025-07-17T11:33:50.382ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch = Switch0
39272025-07-17T11:33:50.382ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
39282025-07-17T11:33:50.389ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "8eee3a3b-2ef8-4225-a084-c71f87566305", "content-length": "682", "date": "Thu, 17 Jul 2025 11:33:50 GMT"} })
39292025-07-17T11:33:50.389ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
39302025-07-17T11:33:50.389ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
39312025-07-17T11:33:50.392ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
39322025-07-17T11:33:50.392ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
39332025-07-17T11:33:50.396ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
39342025-07-17T11:33:50.396ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
39352025-07-17T11:33:50.398ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
39362025-07-17T11:33:50.398ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
39372025-07-17T11:33:50.405ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 640.665066ms iteration = 17 status = {}
39382025-07-17T11:33:51.693ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 395 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = 8d78f7d0-3fe7-4257-8017-7f215dc72b06 response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
39392025-07-17T11:33:54.764ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 98 reason = Timeout
39402025-07-17T11:33:54.764ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
39412025-07-17T11:33:54.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
39422025-07-17T11:33:54.766ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39432025-07-17T11:33:54.768ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
39442025-07-17T11:33:54.768ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
39452025-07-17T11:33:54.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
39462025-07-17T11:33:54.770ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
39472025-07-17T11:33:54.770ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 6.80081ms iteration = 98 status = {"chicken_switches_updated":false}
39482025-07-17T11:33:59.764ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 50 reason = Timeout
39492025-07-17T11:33:59.764ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
39502025-07-17T11:33:59.764ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 50 reason = Timeout
39512025-07-17T11:33:59.764ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 99 reason = Timeout
39522025-07-17T11:33:59.764ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
39532025-07-17T11:33:59.764ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
39542025-07-17T11:33:59.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
39552025-07-17T11:33:59.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
39562025-07-17T11:33:59.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
39572025-07-17T11:33:59.766ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39582025-07-17T11:33:59.766ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39592025-07-17T11:33:59.767ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
39602025-07-17T11:33:59.768ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
39612025-07-17T11:33:59.769ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
39622025-07-17T11:33:59.769ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
39632025-07-17T11:33:59.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
39642025-07-17T11:33:59.770ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
39652025-07-17T11:33:59.770ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
39662025-07-17T11:33:59.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
39672025-07-17T11:33:59.771ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
39682025-07-17T11:33:59.771ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 6.739329ms iteration = 99 status = {"chicken_switches_updated":false}
39692025-07-17T11:33:59.772ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
39702025-07-17T11:33:59.772ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
39712025-07-17T11:33:59.773ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39722025-07-17T11:33:59.776ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
39732025-07-17T11:33:59.776ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
39742025-07-17T11:33:59.776ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
39752025-07-17T11:33:59.790ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
39762025-07-17T11:33:59.790ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 25.621828ms iteration = 50 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
39772025-07-17T11:33:59.805ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 41.194883ms iteration = 50 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
39782025-07-17T11:34:01.692ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 324 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = f2a3dc19-7cb8-41f9-9b1e-93e948b078de response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
39792025-07-17T11:34:04.764ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 100 reason = Timeout
39802025-07-17T11:34:04.764ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
39812025-07-17T11:34:04.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
39822025-07-17T11:34:04.766ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39832025-07-17T11:34:04.769ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
39842025-07-17T11:34:04.769ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
39852025-07-17T11:34:04.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
39862025-07-17T11:34:04.771ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
39872025-07-17T11:34:04.771ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 6.592207ms iteration = 100 status = {"chicken_switches_updated":false}
39882025-07-17T11:34:09.764ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 51 reason = Timeout
39892025-07-17T11:34:09.764ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 51 reason = Timeout
39902025-07-17T11:34:09.764ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
39912025-07-17T11:34:09.764ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 101 reason = Timeout
39922025-07-17T11:34:09.764ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
39932025-07-17T11:34:09.764ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
39942025-07-17T11:34:09.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
39952025-07-17T11:34:09.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
39962025-07-17T11:34:09.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
39972025-07-17T11:34:09.766ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39982025-07-17T11:34:09.767ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39992025-07-17T11:34:09.769ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
40002025-07-17T11:34:09.769ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
40012025-07-17T11:34:09.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
40022025-07-17T11:34:09.770ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
40032025-07-17T11:34:09.770ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
40042025-07-17T11:34:09.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
40052025-07-17T11:34:09.771ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
40062025-07-17T11:34:09.771ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 7.265446ms iteration = 101 status = {"chicken_switches_updated":false}
40072025-07-17T11:34:09.772ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
40082025-07-17T11:34:09.772ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
40092025-07-17T11:34:09.773ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40102025-07-17T11:34:09.776ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
40112025-07-17T11:34:09.776ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
40122025-07-17T11:34:09.776ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
40132025-07-17T11:34:09.788ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
40142025-07-17T11:34:09.788ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 24.624634ms iteration = 51 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
40152025-07-17T11:34:09.799ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
40162025-07-17T11:34:09.799ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
40172025-07-17T11:34:09.800ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 36.468801ms iteration = 51 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
40182025-07-17T11:34:11.693ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 341 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = 87af6a42-3aef-4a67-85ba-2910616a6cf9 response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
40192025-07-17T11:34:14.765ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 102 reason = Timeout
40202025-07-17T11:34:14.765ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
40212025-07-17T11:34:14.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
40222025-07-17T11:34:14.766ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40232025-07-17T11:34:14.769ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
40242025-07-17T11:34:14.769ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
40252025-07-17T11:34:14.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
40262025-07-17T11:34:14.771ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
40272025-07-17T11:34:14.771ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 6.245953ms iteration = 102 status = {"chicken_switches_updated":false}
40282025-07-17T11:34:19.676ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 18 reason = Timeout
40292025-07-17T11:34:19.676ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
40302025-07-17T11:34:19.676ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
40312025-07-17T11:34:19.678ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3853, tv_nsec: 430366350 } })
40322025-07-17T11:34:19.678ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1186
40332025-07-17T11:34:19.708ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1194 zone_address = fd00:1122:3344:101::2
40342025-07-17T11:34:19.708ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
40352025-07-17T11:34:19.708ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "6fafd7fd-4516-4c75-963b-43754e6eb651", "content-length": "26", "date": "Thu, 17 Jul 2025 11:34:19 GMT"} })
40362025-07-17T11:34:19.708ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1197 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
40372025-07-17T11:34:19.709ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
40382025-07-17T11:34:19.733ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 18 reason = Timeout
40392025-07-17T11:34:19.733ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 18 reason = Timeout
40402025-07-17T11:34:19.733ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
40412025-07-17T11:34:19.733ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
40422025-07-17T11:34:19.736ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3853, tv_nsec: 488259534 } })
40432025-07-17T11:34:19.736ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1186
40442025-07-17T11:34:19.736ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.465825ms iteration = 18 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
40452025-07-17T11:34:19.739ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
40462025-07-17T11:34:19.739ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
40472025-07-17T11:34:19.742ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 65.736776ms iteration = 18 status = null
40482025-07-17T11:34:19.764ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 18 reason = Timeout
40492025-07-17T11:34:19.764ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
40502025-07-17T11:34:19.764ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 18 reason = Timeout
40512025-07-17T11:34:19.764ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 18 reason = Timeout
40522025-07-17T11:34:19.764ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
40532025-07-17T11:34:19.764ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
40542025-07-17T11:34:19.764ZDEBGnexus (BackgroundTasks): activating background_task = sp_ereport_ingester iteration = 18 reason = Timeout
40552025-07-17T11:34:19.764ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 18 reason = Timeout
40562025-07-17T11:34:19.764ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
40572025-07-17T11:34:19.764ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 18 reason = Timeout
40582025-07-17T11:34:19.764ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 18 reason = Timeout
40592025-07-17T11:34:19.764ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
40602025-07-17T11:34:19.764ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
40612025-07-17T11:34:19.764ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
40622025-07-17T11:34:19.764ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
40632025-07-17T11:34:19.764ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 18 reason = Timeout
40642025-07-17T11:34:19.764ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
40652025-07-17T11:34:19.764ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 52 reason = Timeout
40662025-07-17T11:34:19.764ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 18 reason = Timeout
40672025-07-17T11:34:19.764ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
40682025-07-17T11:34:19.764ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
40692025-07-17T11:34:19.764ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 18 reason = Timeout
40702025-07-17T11:34:19.764ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
40712025-07-17T11:34:19.764ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 52 reason = Timeout
40722025-07-17T11:34:19.764ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
40732025-07-17T11:34:19.765ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 18 reason = Timeout
40742025-07-17T11:34:19.765ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
40752025-07-17T11:34:19.765ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 18 reason = Timeout
40762025-07-17T11:34:19.765ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 18 reason = Timeout
40772025-07-17T11:34:19.765ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
40782025-07-17T11:34:19.765ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
40792025-07-17T11:34:19.765ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 18 reason = Timeout
40802025-07-17T11:34:19.765ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
40812025-07-17T11:34:19.765ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 103 reason = Timeout
40822025-07-17T11:34:19.765ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
40832025-07-17T11:34:19.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
40842025-07-17T11:34:19.765ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 19 reason = Timeout
40852025-07-17T11:34:19.765ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
40862025-07-17T11:34:19.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
40872025-07-17T11:34:19.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
40882025-07-17T11:34:19.765ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12225, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3853, tv_nsec: 517351767 } })
40892025-07-17T11:34:19.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
40902025-07-17T11:34:19.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
40912025-07-17T11:34:19.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
40922025-07-17T11:34:19.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
40932025-07-17T11:34:19.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
40942025-07-17T11:34:19.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
40952025-07-17T11:34:19.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
40962025-07-17T11:34:19.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
40972025-07-17T11:34:19.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
40982025-07-17T11:34:19.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
40992025-07-17T11:34:19.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
41002025-07-17T11:34:19.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
41012025-07-17T11:34:19.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
41022025-07-17T11:34:19.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
41032025-07-17T11:34:19.768ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
41042025-07-17T11:34:19.768ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1194 zone_address = fd00:1122:3344:101::2
41052025-07-17T11:34:19.768ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
41062025-07-17T11:34:19.768ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
41072025-07-17T11:34:19.768ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41082025-07-17T11:34:19.768ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa92988f-bc77-4352-b8c0-f175cf06875e", "content-length": "26", "date": "Thu, 17 Jul 2025 11:34:19 GMT"} })
41092025-07-17T11:34:19.768ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1197 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
41102025-07-17T11:34:19.769ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41112025-07-17T11:34:19.769ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41122025-07-17T11:34:19.770ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
41132025-07-17T11:34:19.770ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41142025-07-17T11:34:19.770ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
41152025-07-17T11:34:19.770ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
41162025-07-17T11:34:19.770ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
41172025-07-17T11:34:19.771ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41182025-07-17T11:34:19.771ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
41192025-07-17T11:34:19.772ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
41202025-07-17T11:34:19.772ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
41212025-07-17T11:34:19.773ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
41222025-07-17T11:34:19.774ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
41232025-07-17T11:34:19.774ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
41242025-07-17T11:34:19.774ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
41252025-07-17T11:34:19.774ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
41262025-07-17T11:34:19.774ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
41272025-07-17T11:34:19.774ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
41282025-07-17T11:34:19.774ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
41292025-07-17T11:34:19.774ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
41302025-07-17T11:34:19.774ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
41312025-07-17T11:34:19.774ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
41322025-07-17T11:34:19.775ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
41332025-07-17T11:34:19.775ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
41342025-07-17T11:34:19.775ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41352025-07-17T11:34:19.777ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3853, tv_nsec: 529660679 } })
41362025-07-17T11:34:19.778ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1186 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
41372025-07-17T11:34:19.779ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
41382025-07-17T11:34:19.779ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
41392025-07-17T11:34:19.779ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
41402025-07-17T11:34:19.787ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
41412025-07-17T11:34:19.787ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
41422025-07-17T11:34:19.789ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
41432025-07-17T11:34:19.790ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
41442025-07-17T11:34:19.790ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 25.343174ms iteration = 18 status = {"errors":[],"requests_created_ok":[]}
41452025-07-17T11:34:19.790ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
41462025-07-17T11:34:19.790ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
41472025-07-17T11:34:19.856ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 91.567577ms iteration = 18 status = {"errors":[],"finish_invoked_ok":[]}
41482025-07-17T11:34:19.859ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41492025-07-17T11:34:19.862ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
41502025-07-17T11:34:19.862ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
41512025-07-17T11:34:19.863ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
41522025-07-17T11:34:19.863ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
41532025-07-17T11:34:19.863ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
41542025-07-17T11:34:19.863ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
41552025-07-17T11:34:19.863ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/all-sp-ids
41562025-07-17T11:34:19.863ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
41572025-07-17T11:34:19.864ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/all-sp-ids", status: 200, headers: {"content-type": "application/json", "x-request-id": "c49ed36d-7245-486b-925a-2e5f3b11ffc0", "content-length": "78", "date": "Thu, 17 Jul 2025 11:34:19 GMT"} })
41582025-07-17T11:34:19.864ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = switch
41592025-07-17T11:34:19.864ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = sled
41602025-07-17T11:34:19.864ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 1 sp_type = sled
41612025-07-17T11:34:19.864ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = sled
41622025-07-17T11:34:19.864ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 1 sp_type = sled
41632025-07-17T11:34:19.864ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = switch
41642025-07-17T11:34:19.865ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
41652025-07-17T11:34:19.865ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
41662025-07-17T11:34:19.865ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 101.070203ms iteration = 18 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
41672025-07-17T11:34:19.867ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41682025-07-17T11:34:19.867ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 102.510981ms iteration = 52 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
41692025-07-17T11:34:19.868ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 0 sp_type = switch
41702025-07-17T11:34:19.868ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
41712025-07-17T11:34:19.868ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
41722025-07-17T11:34:19.868ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
41732025-07-17T11:34:19.869ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
41742025-07-17T11:34:19.869ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41752025-07-17T11:34:19.872ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
41762025-07-17T11:34:19.872ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
41772025-07-17T11:34:19.872ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
41782025-07-17T11:34:19.873ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 0 sp_type = switch
41792025-07-17T11:34:19.873ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = switch
41802025-07-17T11:34:19.873ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = switch
41812025-07-17T11:34:19.873ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
41822025-07-17T11:34:19.873ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
41832025-07-17T11:34:19.874ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
41842025-07-17T11:34:19.876ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41852025-07-17T11:34:19.876ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 0 sp_type = sled
41862025-07-17T11:34:19.877ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
41872025-07-17T11:34:19.877ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
41882025-07-17T11:34:19.878ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:550 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
41892025-07-17T11:34:19.878ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 113.418345ms iteration = 18 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
41902025-07-17T11:34:19.880ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
41912025-07-17T11:34:19.880ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 115.992069ms iteration = 18 status = {"errors":[],"garbage_collect_requested":[]}
41922025-07-17T11:34:19.881ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 0 sp_type = sled
41932025-07-17T11:34:19.881ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = sled
41942025-07-17T11:34:19.881ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
41952025-07-17T11:34:19.881ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = sled
41962025-07-17T11:34:19.882ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
41972025-07-17T11:34:19.882ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
41982025-07-17T11:34:19.882ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
41992025-07-17T11:34:19.882ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 149.157426ms iteration = 18 status = {}
42002025-07-17T11:34:19.886ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42012025-07-17T11:34:19.887ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
42022025-07-17T11:34:19.887ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 122.206761ms iteration = 103 status = {"chicken_switches_updated":false}
42032025-07-17T11:34:19.888ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1194 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 zone_address = fd00:1122:3344:101::2
42042025-07-17T11:34:19.888ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
42052025-07-17T11:34:19.889ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "2acca11e-dcb5-48c5-be69-3a3fb164210a", "content-length": "26", "date": "Thu, 17 Jul 2025 11:34:19 GMT"} })
42062025-07-17T11:34:19.889ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1197 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
42072025-07-17T11:34:19.891ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
42082025-07-17T11:34:19.893ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
42092025-07-17T11:34:19.893ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
42102025-07-17T11:34:19.893ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
42112025-07-17T11:34:19.901ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42122025-07-17T11:34:19.902ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
42132025-07-17T11:34:19.904ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
42142025-07-17T11:34:19.906ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
42152025-07-17T11:34:19.906ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
42162025-07-17T11:34:19.906ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
42172025-07-17T11:34:19.907ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
42182025-07-17T11:34:19.907ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
42192025-07-17T11:34:19.933ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
42202025-07-17T11:34:19.933ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 168.986788ms iteration = 52 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
42212025-07-17T11:34:19.951ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 186.673071ms iteration = 18 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
42222025-07-17T11:34:19.954ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 1 sp_type = sled
42232025-07-17T11:34:19.956ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 0 sp_type = switch start_ena = None
42242025-07-17T11:34:19.956ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
42252025-07-17T11:34:19.956ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f24c558-a459-467e-847b-575eb57b64b3", "content-length": "81", "date": "Thu, 17 Jul 2025 11:34:19 GMT"} })
42262025-07-17T11:34:19.958ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 1 sp_type = sled
42272025-07-17T11:34:19.958ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 1 sp_type = sled
42282025-07-17T11:34:19.958ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 1 sp_type = sled
42292025-07-17T11:34:19.960ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
42302025-07-17T11:34:19.960ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
42312025-07-17T11:34:19.960ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 195.180563ms iteration = 18 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
42322025-07-17T11:34:19.961ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 1 sp_type = sled start_ena = None
42332025-07-17T11:34:19.961ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/1/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
42342025-07-17T11:34:19.961ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/1/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "938d1cb3-3fa6-4493-a05f-99601079ba60", "content-length": "81", "date": "Thu, 17 Jul 2025 11:34:19 GMT"} })
42352025-07-17T11:34:19.962ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
42362025-07-17T11:34:19.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
42372025-07-17T11:34:19.968ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:372
42382025-07-17T11:34:19.968ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 203.757957ms iteration = 18 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
42392025-07-17T11:34:19.969ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 204.063691ms iteration = 19 status = {"physical_disks_added":0}
42402025-07-17T11:34:19.973ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 0 sp_type = sled start_ena = None
42412025-07-17T11:34:19.973ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
42422025-07-17T11:34:19.973ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c58e8fe-eaa8-40f6-a9be-4f68c37de769", "content-length": "81", "date": "Thu, 17 Jul 2025 11:34:19 GMT"} })
42432025-07-17T11:34:19.976ZDEBGnexus (BackgroundTasks): activation complete background_task = sp_ereport_ingester elapsed = 211.60181ms iteration = 18 status = {"errors":[],"sps":[]}
42442025-07-17T11:34:19.981ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42452025-07-17T11:34:19.985ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 0ae78f24-48db-4cf7-91cb-8d5311f2a17f, lookup_type: ById(0ae78f24-48db-4cf7-91cb-8d5311f2a17f) } result = Ok(())
42462025-07-17T11:34:19.985ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
42472025-07-17T11:34:19.985ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
42482025-07-17T11:34:19.988ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 223.975723ms iteration = 18 status = 0
42492025-07-17T11:34:20.028ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 263.443273ms iteration = 18 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
42502025-07-17T11:34:20.030ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
42512025-07-17T11:34:20.031ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
42522025-07-17T11:34:20.034ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42532025-07-17T11:34:20.036ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
42542025-07-17T11:34:20.036ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
42552025-07-17T11:34:20.036ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
42562025-07-17T11:34:20.038ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42572025-07-17T11:34:20.040ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
42582025-07-17T11:34:20.040ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
42592025-07-17T11:34:20.040ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
42602025-07-17T11:34:20.041ZINFOnexus (BackgroundTasks): VPC route manager sled 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
42612025-07-17T11:34:20.041ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
42622025-07-17T11:34:20.042ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c483fdf-c2f4-4f15-b4fd-d4c569189ed0", "content-length": "372", "date": "Thu, 17 Jul 2025 11:34:20 GMT"} })
42632025-07-17T11:34:20.042ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
42642025-07-17T11:34:20.042ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
42652025-07-17T11:34:20.070ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
42662025-07-17T11:34:20.070ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
42672025-07-17T11:34:20.070ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "2bc4b323-d819-43aa-bcdb-48548c0c34dc", "date": "Thu, 17 Jul 2025 11:34:20 GMT"} })
42682025-07-17T11:34:20.070ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
42692025-07-17T11:34:20.071ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
42702025-07-17T11:34:20.075ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
42712025-07-17T11:34:20.075ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
42722025-07-17T11:34:20.076ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
42732025-07-17T11:34:20.076ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
42742025-07-17T11:34:20.078ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 151caa7c-6a05-4995-94b4-7946d428117d, rack_id: 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5, switch_location: "switch0", port_name: Name(Name("qsfp0")), port_settings_id: Some(2e30c7ae-26da-46e4-9127-aac194921f20) } rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0
42752025-07-17T11:34:20.078ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
42762025-07-17T11:34:20.079ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
42772025-07-17T11:34:20.079ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
42782025-07-17T11:34:20.080ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
42792025-07-17T11:34:20.083ZINFOnexus (BackgroundTasks): VPC route manager sled 0ae78f24-48db-4cf7-91cb-8d5311f2a17f push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
42802025-07-17T11:34:20.083ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
42812025-07-17T11:34:20.083ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "9b164d90-84ef-4755-8b1b-2d48f0e6e127", "date": "Thu, 17 Jul 2025 11:34:20 GMT"} })
42822025-07-17T11:34:20.085ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 320.694517ms iteration = 18 status = {}
42832025-07-17T11:34:20.093ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1832 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = Name(Name("qsfp0"))
42842025-07-17T11:34:20.093ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1859 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0 switch_port_id = Name(Name("qsfp0")) switch_port_settings_id = 2e30c7ae-26da-46e4-9127-aac194921f20
42852025-07-17T11:34:20.093ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
42862025-07-17T11:34:20.093ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
42872025-07-17T11:34:20.093ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
42882025-07-17T11:34:20.093ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
42892025-07-17T11:34:20.093ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
42902025-07-17T11:34:20.093ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2006 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
42912025-07-17T11:34:20.094ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
42922025-07-17T11:34:20.094ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2042 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
42932025-07-17T11:34:20.094ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
42942025-07-17T11:34:20.094ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
42952025-07-17T11:34:20.094ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
42962025-07-17T11:34:20.097ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0
42972025-07-17T11:34:20.097ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
42982025-07-17T11:34:20.175ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
42992025-07-17T11:34:20.175ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef623cd3-36f4-40f8-86a6-15a0b5a39f72", "content-length": "691", "date": "Thu, 17 Jul 2025 11:34:20 GMT"} })
43002025-07-17T11:34:20.175ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
43012025-07-17T11:34:20.175ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
43022025-07-17T11:34:20.175ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
43032025-07-17T11:34:20.176ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 412.129623ms iteration = 18 status = {}
43042025-07-17T11:34:20.234ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "16a49b90-9cc4-43d4-a96e-4b9c8038e7d8", "date": "Thu, 17 Jul 2025 11:34:20 GMT"} })
43052025-07-17T11:34:20.234ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch = Switch0
43062025-07-17T11:34:20.234ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
43072025-07-17T11:34:20.241ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "62be0462-b025-44bb-87a0-a0ccbe298a4e", "content-length": "682", "date": "Thu, 17 Jul 2025 11:34:20 GMT"} })
43082025-07-17T11:34:20.241ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
43092025-07-17T11:34:20.241ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
43102025-07-17T11:34:20.245ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
43112025-07-17T11:34:20.245ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
43122025-07-17T11:34:20.248ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
43132025-07-17T11:34:20.249ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
43142025-07-17T11:34:20.251ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
43152025-07-17T11:34:20.251ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
43162025-07-17T11:34:20.257ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 492.937279ms iteration = 18 status = {}
43172025-07-17T11:34:20.529ZDEBGnexus (producer-registration-task): registering as oximeter producer with Nexus registration_count = 3
43182025-07-17T11:34:20.529ZDEBGnexus (producer-registration-task): will register with Nexus at [fd00:1122:3344:101::b]:12221
43192025-07-17T11:34:20.559ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::b]:12221/metrics/producers
43202025-07-17T11:34:20.559ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::b]:49380
43212025-07-17T11:34:20.559ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:49380 req_id = 6d84ae9c-6e5a-429e-af24-e5c0e5ae7280 roles = RoleSet { roles: {} } uri = /metrics/producers
43222025-07-17T11:34:20.560ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:49380 req_id = 6d84ae9c-6e5a-429e-af24-e5c0e5ae7280 resource = Database result = Ok(()) uri = /metrics/producers
43232025-07-17T11:34:20.598ZINFOnexus: assigned collector to new producer collector_id = 3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d file = nexus/src/app/oximeter.rs:76 producer_id = 5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
43242025-07-17T11:34:20.598ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 39091 local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:49380 req_id = 6d84ae9c-6e5a-429e-af24-e5c0e5ae7280 response_code = 201 uri = /metrics/producers
43252025-07-17T11:34:20.599ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "6d84ae9c-6e5a-429e-af24-e5c0e5ae7280", "content-length": "41", "date": "Thu, 17 Jul 2025 11:34:20 GMT"} })
43262025-07-17T11:34:20.601ZDEBGnexus (producer-registration-task): successfully registered with Nexus, pausing until time to renew lease lease_duration = 600s wait_period = 150s
43272025-07-17T11:34:21.692ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 371 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = a0388ce0-0271-4df3-82b8-473e382a3e20 response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
43282025-07-17T11:34:24.764ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 104 reason = Timeout
43292025-07-17T11:34:24.764ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
43302025-07-17T11:34:24.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
43312025-07-17T11:34:24.766ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43322025-07-17T11:34:24.769ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
43332025-07-17T11:34:24.769ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
43342025-07-17T11:34:24.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
43352025-07-17T11:34:24.771ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
43362025-07-17T11:34:24.771ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 7.026183ms iteration = 104 status = {"chicken_switches_updated":false}
43372025-07-17T11:34:29.765ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 53 reason = Timeout
43382025-07-17T11:34:29.765ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
43392025-07-17T11:34:29.765ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 53 reason = Timeout
43402025-07-17T11:34:29.765ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 105 reason = Timeout
43412025-07-17T11:34:29.765ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
43422025-07-17T11:34:29.765ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
43432025-07-17T11:34:29.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
43442025-07-17T11:34:29.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
43452025-07-17T11:34:29.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
43462025-07-17T11:34:29.766ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
43472025-07-17T11:34:29.767ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43482025-07-17T11:34:29.767ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
43492025-07-17T11:34:29.767ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43502025-07-17T11:34:29.768ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.501596ms iteration = 53 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
43512025-07-17T11:34:29.770ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
43522025-07-17T11:34:29.770ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
43532025-07-17T11:34:29.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
43542025-07-17T11:34:29.771ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
43552025-07-17T11:34:29.771ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
43562025-07-17T11:34:29.771ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
43572025-07-17T11:34:29.772ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
43582025-07-17T11:34:29.772ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 7.081863ms iteration = 105 status = {"chicken_switches_updated":false}
43592025-07-17T11:34:29.773ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
43602025-07-17T11:34:29.773ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
43612025-07-17T11:34:29.774ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43622025-07-17T11:34:29.777ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
43632025-07-17T11:34:29.777ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
43642025-07-17T11:34:29.777ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
43652025-07-17T11:34:29.789ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
43662025-07-17T11:34:29.789ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 24.067797ms iteration = 53 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
43672025-07-17T11:34:31.692ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 528 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = 1b06ffdd-aff7-4da6-ab6f-afeee90f2dc7 response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
43682025-07-17T11:34:34.764ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 106 reason = Timeout
43692025-07-17T11:34:34.764ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
43702025-07-17T11:34:34.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
43712025-07-17T11:34:34.766ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43722025-07-17T11:34:34.770ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
43732025-07-17T11:34:34.770ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
43742025-07-17T11:34:34.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
43752025-07-17T11:34:34.772ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
43762025-07-17T11:34:34.772ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 7.561949ms iteration = 106 status = {"chicken_switches_updated":false}
43772025-07-17T11:34:39.764ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 54 reason = Timeout
43782025-07-17T11:34:39.764ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 54 reason = Timeout
43792025-07-17T11:34:39.764ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
43802025-07-17T11:34:39.764ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
43812025-07-17T11:34:39.764ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 107 reason = Timeout
43822025-07-17T11:34:39.764ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
43832025-07-17T11:34:39.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
43842025-07-17T11:34:39.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
43852025-07-17T11:34:39.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
43862025-07-17T11:34:39.767ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43872025-07-17T11:34:39.768ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43882025-07-17T11:34:39.770ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
43892025-07-17T11:34:39.770ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
43902025-07-17T11:34:39.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
43912025-07-17T11:34:39.771ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
43922025-07-17T11:34:39.771ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
43932025-07-17T11:34:39.771ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
43942025-07-17T11:34:39.773ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
43952025-07-17T11:34:39.773ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 8.754495ms iteration = 107 status = {"chicken_switches_updated":false}
43962025-07-17T11:34:39.773ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
43972025-07-17T11:34:39.773ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
43982025-07-17T11:34:39.775ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43992025-07-17T11:34:39.777ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
44002025-07-17T11:34:39.777ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
44012025-07-17T11:34:39.777ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
44022025-07-17T11:34:39.787ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
44032025-07-17T11:34:39.787ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.912462ms iteration = 54 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
44042025-07-17T11:34:39.800ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
44052025-07-17T11:34:39.801ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
44062025-07-17T11:34:39.802ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 38.126083ms iteration = 54 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
44072025-07-17T11:34:41.693ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 328 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = 657798c4-e9b1-447f-886e-cebebf246e2c response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
44082025-07-17T11:34:44.764ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 108 reason = Timeout
44092025-07-17T11:34:44.765ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
44102025-07-17T11:34:44.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
44112025-07-17T11:34:44.767ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44122025-07-17T11:34:44.770ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
44132025-07-17T11:34:44.770ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
44142025-07-17T11:34:44.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
44152025-07-17T11:34:44.772ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
44162025-07-17T11:34:44.772ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 7.467018ms iteration = 108 status = {"chicken_switches_updated":false}
44172025-07-17T11:34:46.685ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::d]:50893
44182025-07-17T11:34:46.685ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:50893 req_id = c8adce4f-bf90-4503-bb81-bd58b5b6c308 roles = RoleSet { roles: {} } uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?sort_by=id_ascending
44192025-07-17T11:34:46.685ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:50893 req_id = c8adce4f-bf90-4503-bb81-bd58b5b6c308 resource = Database result = Ok(()) uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?sort_by=id_ascending
44202025-07-17T11:34:46.688ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 3166 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:50893 req_id = c8adce4f-bf90-4503-bb81-bd58b5b6c308 response_code = 200 uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?sort_by=id_ascending
44212025-07-17T11:34:46.688ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:50893 req_id = 496892bb-ce47-456d-8279-a8f73ccd8888 roles = RoleSet { roles: {} } uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmMzlkM2U3MC02NmU0LTQ2OGEtYTQ0ZC03NDZkMjJiZWEzYTAifX0%3D
44222025-07-17T11:34:46.689ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:50893 req_id = 496892bb-ce47-456d-8279-a8f73ccd8888 resource = Database result = Ok(()) uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmMzlkM2U3MC02NmU0LTQ2OGEtYTQ0ZC03NDZkMjJiZWEzYTAifX0%3D
44232025-07-17T11:34:46.691ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 2488 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:50893 req_id = 496892bb-ce47-456d-8279-a8f73ccd8888 response_code = 200 uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmMzlkM2U3MC02NmU0LTQ2OGEtYTQ0ZC03NDZkMjJiZWEzYTAifX0%3D
44242025-07-17T11:34:49.677ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 10 reason = Timeout
44252025-07-17T11:34:49.677ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 10 reason = Timeout
44262025-07-17T11:34:49.677ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
44272025-07-17T11:34:49.677ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
44282025-07-17T11:34:49.677ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 12 reason = Timeout
44292025-07-17T11:34:49.677ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 10 reason = Timeout
44302025-07-17T11:34:49.677ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-07-17 11:24:49.677323887 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
44312025-07-17T11:34:49.677ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
44322025-07-17T11:34:49.677ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 10 reason = Timeout
44332025-07-17T11:34:49.677ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 10 reason = Timeout
44342025-07-17T11:34:49.677ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
44352025-07-17T11:34:49.677ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
44362025-07-17T11:34:49.677ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
44372025-07-17T11:34:49.677ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 19 reason = Timeout
44382025-07-17T11:34:49.677ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
44392025-07-17T11:34:49.677ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 12 reason = Timeout
44402025-07-17T11:34:49.677ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 10 reason = Timeout
44412025-07-17T11:34:49.677ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
44422025-07-17T11:34:49.677ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
44432025-07-17T11:34:49.677ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
44442025-07-17T11:34:49.677ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
44452025-07-17T11:34:49.677ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
44462025-07-17T11:34:49.677ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("3dd3bf3b-7760-4750-b1d3-52ee777335f0.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("79e5c77d-18b3-4951-8d11-cfa3eae2b3e3.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("79e5c77d-18b3-4951-8d11-cfa3eae2b3e3.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::8)) }], valid_until: Instant { tv_sec: 3883, tv_nsec: 429786820 } })
44472025-07-17T11:34:49.677ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("3144a75a-317f-4b68-93e2-a210ca1189d1.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("87408888-9df4-4e0c-bc8e-2f920d039100.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("ee4378c5-e5f2-41cf-8969-e5e03e1b9a41.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("ee4378c5-e5f2-41cf-8969-e5e03e1b9a41.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:3::1)) }], valid_until: Instant { tv_sec: 3883, tv_nsec: 429841590 } })
44482025-07-17T11:34:49.678ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
44492025-07-17T11:34:49.678ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.246557ms iteration = 10 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
44502025-07-17T11:34:49.678ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } dns_group = internal
44512025-07-17T11:34:49.678ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.129315ms iteration = 10 status = {"addresses":["[fd00:1122:3344:1::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:3::1]:5353"]}
44522025-07-17T11:34:49.679ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
44532025-07-17T11:34:49.679ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 11 reason = Timeout
44542025-07-17T11:34:49.679ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:85 reason = blueprint disabled target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a
44552025-07-17T11:34:49.679ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 20.85\u{b5}s iteration = 11 status = {"enabled":false,"target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a"}
44562025-07-17T11:34:49.679ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44572025-07-17T11:34:49.679ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
44582025-07-17T11:34:49.679ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
44592025-07-17T11:34:49.680ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
44602025-07-17T11:34:49.680ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3883, tv_nsec: 432482575 } })
44612025-07-17T11:34:49.680ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1186
44622025-07-17T11:34:49.682ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
44632025-07-17T11:34:49.682ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
44642025-07-17T11:34:49.682ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
44652025-07-17T11:34:49.683ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44662025-07-17T11:34:49.686ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44672025-07-17T11:34:49.686ZDEBGnexus (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 5 reason = Timeout
44682025-07-17T11:34:49.686ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
44692025-07-17T11:34:49.686ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 10 reason = Timeout
44702025-07-17T11:34:49.686ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:53
44712025-07-17T11:34:49.686ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 10 reason = Timeout
44722025-07-17T11:34:49.686ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
44732025-07-17T11:34:49.686ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
44742025-07-17T11:34:49.686ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_planner iteration = 12 reason = Timeout
44752025-07-17T11:34:49.686ZDEBGnexus (BackgroundTasks): blueprint planning disabled, doing nothing background_task = blueprint_planner
44762025-07-17T11:34:49.686ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_planner elapsed = 9.36\u{b5}s iteration = 12 status = "Disabled"
44772025-07-17T11:34:49.686ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 10 reason = Timeout
44782025-07-17T11:34:49.686ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
44792025-07-17T11:34:49.686ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
44802025-07-17T11:34:49.686ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
44812025-07-17T11:34:49.686ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
44822025-07-17T11:34:49.686ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
44832025-07-17T11:34:49.686ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
44842025-07-17T11:34:49.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
44852025-07-17T11:34:49.687ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
44862025-07-17T11:34:49.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
44872025-07-17T11:34:49.687ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
44882025-07-17T11:34:49.687ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-07-17 11:24:49.677323887 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
44892025-07-17T11:34:49.687ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 10.148874ms iteration = 10 status = {"expiration":"2025-07-17T11:24:49.677323887Z","pruned":[]}
44902025-07-17T11:34:49.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
44912025-07-17T11:34:49.687ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
44922025-07-17T11:34:49.687ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
44932025-07-17T11:34:49.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
44942025-07-17T11:34:49.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
44952025-07-17T11:34:49.688ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
44962025-07-17T11:34:49.689ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44972025-07-17T11:34:49.690ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
44982025-07-17T11:34:49.692ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44992025-07-17T11:34:49.692ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 10 reason = Timeout
45002025-07-17T11:34:49.692ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
45012025-07-17T11:34:49.692ZDEBGnexus (BackgroundTasks): activating background_task = webhook_deliverator iteration = 10 reason = Timeout
45022025-07-17T11:34:49.692ZDEBGnexus (BackgroundTasks): activating background_task = alert_dispatcher iteration = 10 reason = Timeout
45032025-07-17T11:34:49.692ZDEBGnexus (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} }
45042025-07-17T11:34:49.692ZDEBGnexus (BackgroundTasks): roles background_task = alert_dispatcher roles = RoleSet { roles: {} }
45052025-07-17T11:34:49.693ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
45062025-07-17T11:34:49.693ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = alert_dispatcher resource = Database result = Ok(())
45072025-07-17T11:34:49.693ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database result = Ok(())
45082025-07-17T11:34:49.693ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
45092025-07-17T11:34:49.693ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
45102025-07-17T11:34:49.693ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
45112025-07-17T11:34:49.694ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
45122025-07-17T11:34:49.694ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
45132025-07-17T11:34:49.694ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
45142025-07-17T11:34:49.701ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45152025-07-17T11:34:49.704ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
45162025-07-17T11:34:49.704ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
45172025-07-17T11:34:49.705ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
45182025-07-17T11:34:49.705ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45192025-07-17T11:34:49.706ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45202025-07-17T11:34:49.708ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 15.343272ms iteration = 10 status = {"errors":[],"found_port_ok":[]}
45212025-07-17T11:34:49.708ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
45222025-07-17T11:34:49.708ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
45232025-07-17T11:34:49.708ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
45242025-07-17T11:34:49.708ZINFOnexus (BackgroundTasks): all webhook delivery tasks completed background_task = webhook_deliverator file = nexus/src/app/background/tasks/webhook_deliverator.rs:204 num_receivers = 0
45252025-07-17T11:34:49.708ZDEBGnexus (BackgroundTasks): activation complete background_task = webhook_deliverator elapsed = 16.050452ms iteration = 10 status = {"by_rx":{},"error":null}
45262025-07-17T11:34:49.710ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45272025-07-17T11:34:49.711ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
45282025-07-17T11:34:49.711ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
45292025-07-17T11:34:49.711ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
45302025-07-17T11:34:49.713ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
45312025-07-17T11:34:49.713ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
45322025-07-17T11:34:49.713ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
45332025-07-17T11:34:49.721ZDEBGnexus (BackgroundTasks): roles background_task = alert_dispatcher roles = RoleSet { roles: {} }
45342025-07-17T11:34:49.722ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = alert_dispatcher resource = Database result = Ok(())
45352025-07-17T11:34:49.727ZDEBGnexus (BackgroundTasks): activation complete background_task = alert_dispatcher elapsed = 34.90569ms iteration = 10 status = {"dispatched":[],"errors":[],"glob_version":"161.0.0","globs_reprocessed":{},"no_receivers":[]}
45362025-07-17T11:34:49.734ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 19 reason = Timeout
45372025-07-17T11:34:49.734ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 19 reason = Timeout
45382025-07-17T11:34:49.734ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
45392025-07-17T11:34:49.734ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
45402025-07-17T11:34:49.737ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3883, tv_nsec: 489312474 } })
45412025-07-17T11:34:49.737ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1186
45422025-07-17T11:34:49.739ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.943135ms iteration = 19 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
45432025-07-17T11:34:49.753ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
45442025-07-17T11:34:49.753ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:170 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
45452025-07-17T11:34:49.753ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 66.78907ms iteration = 10 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
45462025-07-17T11:34:49.755ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1194 zone_address = fd00:1122:3344:101::2
45472025-07-17T11:34:49.755ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
45482025-07-17T11:34:49.755ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
45492025-07-17T11:34:49.755ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "4eb38f83-dbce-4533-abac-9847db77296a", "content-length": "26", "date": "Thu, 17 Jul 2025 11:34:49 GMT"} })
45502025-07-17T11:34:49.755ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1197 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
45512025-07-17T11:34:49.758ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
45522025-07-17T11:34:49.758ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
45532025-07-17T11:34:49.758ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
45542025-07-17T11:34:49.758ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
45552025-07-17T11:34:49.758ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
45562025-07-17T11:34:49.759ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
45572025-07-17T11:34:49.761ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45582025-07-17T11:34:49.761ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
45592025-07-17T11:34:49.762ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
45602025-07-17T11:34:49.762ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
45612025-07-17T11:34:49.762ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
45622025-07-17T11:34:49.762ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
45632025-07-17T11:34:49.762ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
45642025-07-17T11:34:49.763ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
45652025-07-17T11:34:49.763ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45662025-07-17T11:34:49.764ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45672025-07-17T11:34:49.764ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 19 reason = Timeout
45682025-07-17T11:34:49.764ZDEBGnexus (BackgroundTasks): activating background_task = sp_ereport_ingester iteration = 19 reason = Timeout
45692025-07-17T11:34:49.764ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
45702025-07-17T11:34:49.764ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 19 reason = Timeout
45712025-07-17T11:34:49.764ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
45722025-07-17T11:34:49.764ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 19 reason = Timeout
45732025-07-17T11:34:49.764ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 19 reason = Timeout
45742025-07-17T11:34:49.764ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
45752025-07-17T11:34:49.764ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 19 reason = Timeout
45762025-07-17T11:34:49.764ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 19 reason = Timeout
45772025-07-17T11:34:49.764ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
45782025-07-17T11:34:49.764ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
45792025-07-17T11:34:49.764ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
45802025-07-17T11:34:49.764ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
45812025-07-17T11:34:49.764ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 19 reason = Timeout
45822025-07-17T11:34:49.764ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
45832025-07-17T11:34:49.764ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 55 reason = Timeout
45842025-07-17T11:34:49.764ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
45852025-07-17T11:34:49.764ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 55 reason = Timeout
45862025-07-17T11:34:49.764ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 20 reason = Timeout
45872025-07-17T11:34:49.764ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 19 reason = Timeout
45882025-07-17T11:34:49.764ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
45892025-07-17T11:34:49.765ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
45902025-07-17T11:34:49.765ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
45912025-07-17T11:34:49.765ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 19 reason = Timeout
45922025-07-17T11:34:49.765ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
45932025-07-17T11:34:49.765ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 19 reason = Timeout
45942025-07-17T11:34:49.765ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
45952025-07-17T11:34:49.765ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 19 reason = Timeout
45962025-07-17T11:34:49.765ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
45972025-07-17T11:34:49.765ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 19 reason = Timeout
45982025-07-17T11:34:49.765ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 19 reason = Timeout
45992025-07-17T11:34:49.765ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
46002025-07-17T11:34:49.765ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
46012025-07-17T11:34:49.765ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
46022025-07-17T11:34:49.765ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 109 reason = Timeout
46032025-07-17T11:34:49.765ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
46042025-07-17T11:34:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
46052025-07-17T11:34:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
46062025-07-17T11:34:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
46072025-07-17T11:34:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
46082025-07-17T11:34:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
46092025-07-17T11:34:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
46102025-07-17T11:34:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
46112025-07-17T11:34:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
46122025-07-17T11:34:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
46132025-07-17T11:34:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
46142025-07-17T11:34:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
46152025-07-17T11:34:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
46162025-07-17T11:34:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
46172025-07-17T11:34:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
46182025-07-17T11:34:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
46192025-07-17T11:34:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
46202025-07-17T11:34:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
46212025-07-17T11:34:49.765ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12225, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3883, tv_nsec: 517503826 } })
46222025-07-17T11:34:49.766ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
46232025-07-17T11:34:49.766ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
46242025-07-17T11:34:49.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
46252025-07-17T11:34:49.767ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
46262025-07-17T11:34:49.767ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
46272025-07-17T11:34:49.767ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
46282025-07-17T11:34:49.768ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46292025-07-17T11:34:49.769ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46302025-07-17T11:34:49.770ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46312025-07-17T11:34:49.770ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
46322025-07-17T11:34:49.770ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:138 found = 0 not_ready_to_be_deleted = 0
46332025-07-17T11:34:49.770ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 83.833215ms iteration = 10 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
46342025-07-17T11:34:49.770ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46352025-07-17T11:34:49.771ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
46362025-07-17T11:34:49.771ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
46372025-07-17T11:34:49.771ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
46382025-07-17T11:34:49.772ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46392025-07-17T11:34:49.772ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
46402025-07-17T11:34:49.772ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
46412025-07-17T11:34:49.772ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46422025-07-17T11:34:49.772ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
46432025-07-17T11:34:49.772ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46442025-07-17T11:34:49.773ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46452025-07-17T11:34:49.774ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46462025-07-17T11:34:49.775ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
46472025-07-17T11:34:49.775ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
46482025-07-17T11:34:49.775ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
46492025-07-17T11:34:49.776ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
46502025-07-17T11:34:49.776ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
46512025-07-17T11:34:49.776ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
46522025-07-17T11:34:49.777ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
46532025-07-17T11:34:49.777ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
46542025-07-17T11:34:49.777ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
46552025-07-17T11:34:49.778ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
46562025-07-17T11:34:49.778ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
46572025-07-17T11:34:49.778ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
46582025-07-17T11:34:49.779ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
46592025-07-17T11:34:49.779ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
46602025-07-17T11:34:49.779ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
46612025-07-17T11:34:49.779ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
46622025-07-17T11:34:49.779ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
46632025-07-17T11:34:49.779ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
46642025-07-17T11:34:49.779ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
46652025-07-17T11:34:49.779ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
46662025-07-17T11:34:49.779ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
46672025-07-17T11:34:49.779ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
46682025-07-17T11:34:49.779ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
46692025-07-17T11:34:49.779ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
46702025-07-17T11:34:49.780ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46712025-07-17T11:34:49.782ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
46722025-07-17T11:34:49.782ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 4a9e4c5e-728e-4850-bd58-14548c02bf6f dns_zone_name = oxide.test found_so_far = 0 version = 2
46732025-07-17T11:34:49.782ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
46742025-07-17T11:34:49.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
46752025-07-17T11:34:49.783ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
46762025-07-17T11:34:49.783ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
46772025-07-17T11:34:49.784ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
46782025-07-17T11:34:49.784ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
46792025-07-17T11:34:49.784ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
46802025-07-17T11:34:49.785ZDEBGnexus (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 99.066987ms iteration = 5 status = {"nsuccess":5}
46812025-07-17T11:34:49.786ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
46822025-07-17T11:34:49.786ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
46832025-07-17T11:34:49.788ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
46842025-07-17T11:34:49.788ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
46852025-07-17T11:34:49.791ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3883, tv_nsec: 542954161 } })
46862025-07-17T11:34:49.791ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
46872025-07-17T11:34:49.791ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
46882025-07-17T11:34:49.791ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1186 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
46892025-07-17T11:34:49.794ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
46902025-07-17T11:34:49.794ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 29.425918ms iteration = 109 status = {"chicken_switches_updated":false}
46912025-07-17T11:34:49.797ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46922025-07-17T11:34:49.800ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:550 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
46932025-07-17T11:34:49.800ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 35.608119ms iteration = 19 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
46942025-07-17T11:34:49.802ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46952025-07-17T11:34:49.803ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
46962025-07-17T11:34:49.803ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
46972025-07-17T11:34:49.803ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
46982025-07-17T11:34:49.806ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
46992025-07-17T11:34:49.806ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
47002025-07-17T11:34:49.806ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
47012025-07-17T11:34:49.864ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
47022025-07-17T11:34:49.864ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 99.722715ms iteration = 55 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
47032025-07-17T11:34:49.867ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 102.286228ms iteration = 19 status = {"errors":[],"finish_invoked_ok":[]}
47042025-07-17T11:34:49.869ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
47052025-07-17T11:34:49.870ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
47062025-07-17T11:34:49.877ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 112.798177ms iteration = 19 status = {"errors":[],"garbage_collect_requested":[]}
47072025-07-17T11:34:49.879ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
47082025-07-17T11:34:49.880ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
47092025-07-17T11:34:49.892ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
47102025-07-17T11:34:49.892ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 915fbba0-947f-4bf0-9f9f-48a79df81f7b dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
47112025-07-17T11:34:49.892ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
47122025-07-17T11:34:49.893ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
47132025-07-17T11:34:49.895ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
47142025-07-17T11:34:49.895ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
47152025-07-17T11:34:49.895ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
47162025-07-17T11:34:49.895ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
47172025-07-17T11:34:49.898ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 133.365778ms iteration = 19 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
47182025-07-17T11:34:49.898ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
47192025-07-17T11:34:49.899ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
47202025-07-17T11:34:49.908ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-07-17T11:25:46.964149Z, time_modified: 2025-07-17T11:25:46.964149Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 1fa836d9-7854-4ada-8c89-c26d922646c6, db_silo: Silo { identity: SiloIdentity { id: 1fa836d9-7854-4ada-8c89-c26d922646c6, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-07-17T11:26:35.643175Z, time_modified: 2025-07-17T11:26:35.643175Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "d7c3455b70a66c86ec2a7e597ca46fe0ea0c49dde5c1363f198c20aa77d58442" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 1fa836d9-7854-4ada-8c89-c26d922646c6, db_silo: Silo { identity: SiloIdentity { id: 1fa836d9-7854-4ada-8c89-c26d922646c6, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-07-17T11:26:35.643175Z, time_modified: 2025-07-17T11:26:35.643175Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "d7c3455b70a66c86ec2a7e597ca46fe0ea0c49dde5c1363f198c20aa77d58442" }] }) }
47212025-07-17T11:34:49.908ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 231.059596ms iteration = 10 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"1fa836d9-7854-4ada-8c89-c26d922646c6","tls_certs":["d7c3455b70a66c86ec2a7e597ca46fe0ea0c49dde5c1363f198c20aa77d58442"]}},"default_endpoint":{"silo_id":"1fa836d9-7854-4ada-8c89-c26d922646c6","tls_certs":["d7c3455b70a66c86ec2a7e597ca46fe0ea0c49dde5c1363f198c20aa77d58442"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
47222025-07-17T11:34:49.916ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47232025-07-17T11:34:49.918ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
47242025-07-17T11:34:49.919ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
47252025-07-17T11:34:49.919ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 154.75372ms iteration = 19 status = {"errors":[],"requests_created_ok":[]}
47262025-07-17T11:34:49.920ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
47272025-07-17T11:34:49.920ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
47282025-07-17T11:34:49.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
47292025-07-17T11:34:49.927ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47302025-07-17T11:34:49.932ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
47312025-07-17T11:34:49.932ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
47322025-07-17T11:34:49.933ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
47332025-07-17T11:34:49.935ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 4a9e4c5e-728e-4850-bd58-14548c02bf6f dns_zone_name = oxide.test found_so_far = 4 version = 2
47342025-07-17T11:34:49.935ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
47352025-07-17T11:34:49.936ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-07-17 11:26:35.847594 UTC dns_group = external generation = 2
47362025-07-17T11:34:49.936ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 258.678021ms iteration = 10 status = {"generation":2}
47372025-07-17T11:34:49.949ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 184.618205ms iteration = 20 status = {"physical_disks_added":0}
47382025-07-17T11:34:49.958ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
47392025-07-17T11:34:49.962ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
47402025-07-17T11:34:49.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
47412025-07-17T11:34:49.963ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
47422025-07-17T11:34:49.963ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
47432025-07-17T11:34:49.967ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
47442025-07-17T11:34:49.968ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
47452025-07-17T11:34:49.978ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
47462025-07-17T11:34:49.978ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "ee6aafb1-e7ac-4846-9472-0a6c5a45466a", "date": "Thu, 17 Jul 2025 11:34:49 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
47472025-07-17T11:34:49.978ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
47482025-07-17T11:34:49.981ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47492025-07-17T11:34:49.987ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 0ae78f24-48db-4cf7-91cb-8d5311f2a17f, lookup_type: ById(0ae78f24-48db-4cf7-91cb-8d5311f2a17f) } result = Ok(())
47502025-07-17T11:34:49.987ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
47512025-07-17T11:34:49.987ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
47522025-07-17T11:34:49.987ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
47532025-07-17T11:34:49.987ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
47542025-07-17T11:34:49.988ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
47552025-07-17T11:34:49.988ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
47562025-07-17T11:34:49.988ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
47572025-07-17T11:34:49.988ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
47582025-07-17T11:34:49.990ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 225.584115ms iteration = 19 status = 0
47592025-07-17T11:34:50.000ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47602025-07-17T11:34:50.002ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "3293959c-a856-40c7-b929-796b711b1eb4", "date": "Thu, 17 Jul 2025 11:34:49 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
47612025-07-17T11:34:50.002ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
47622025-07-17T11:34:50.002ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 237.436621ms iteration = 55 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
47632025-07-17T11:34:50.005ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
47642025-07-17T11:34:50.005ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
47652025-07-17T11:34:50.006ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
47662025-07-17T11:34:50.006ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 328.890836ms iteration = 12 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
47672025-07-17T11:34:50.006ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
47682025-07-17T11:34:50.010ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 915fbba0-947f-4bf0-9f9f-48a79df81f7b dns_zone_name = control-plane.oxide.internal found_so_far = 59 version = 1
47692025-07-17T11:34:50.010ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
47702025-07-17T11:34:50.010ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-07-17 11:26:34.922237 UTC dns_group = internal generation = 1
47712025-07-17T11:34:50.010ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 332.913049ms iteration = 10 status = {"generation":1}
47722025-07-17T11:34:50.010ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:372
47732025-07-17T11:34:50.010ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 245.566087ms iteration = 19 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
47742025-07-17T11:34:50.012ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 247.543113ms iteration = 19 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
47752025-07-17T11:34:50.022ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1194 zone_address = fd00:1122:3344:101::2
47762025-07-17T11:34:50.022ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
47772025-07-17T11:34:50.026ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "0afc8146-c6ac-4ba3-a635-db03794f524e", "content-length": "26", "date": "Thu, 17 Jul 2025 11:34:49 GMT"} })
47782025-07-17T11:34:50.026ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1197 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
47792025-07-17T11:34:50.028ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
47802025-07-17T11:34:50.057ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/all-sp-ids
47812025-07-17T11:34:50.058ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/all-sp-ids", status: 200, headers: {"content-type": "application/json", "x-request-id": "518bd5ad-6c3e-4b2d-a344-8456e30b3c0d", "content-length": "78", "date": "Thu, 17 Jul 2025 11:34:49 GMT"} })
47822025-07-17T11:34:50.058ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = switch
47832025-07-17T11:34:50.058ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = sled
47842025-07-17T11:34:50.058ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 1 sp_type = sled
47852025-07-17T11:34:50.058ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = sled
47862025-07-17T11:34:50.058ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = switch
47872025-07-17T11:34:50.058ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 1 sp_type = sled
47882025-07-17T11:34:50.061ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
47892025-07-17T11:34:50.061ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
47902025-07-17T11:34:50.061ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 296.294206ms iteration = 19 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
47912025-07-17T11:34:50.066ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 0 sp_type = switch
47922025-07-17T11:34:50.068ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 0 sp_type = sled
47932025-07-17T11:34:50.069ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 1 sp_type = sled
47942025-07-17T11:34:50.069ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 0 sp_type = switch
47952025-07-17T11:34:50.069ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = switch
47962025-07-17T11:34:50.069ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
47972025-07-17T11:34:50.070ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = switch
47982025-07-17T11:34:50.070ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
47992025-07-17T11:34:50.072ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 0 sp_type = sled
48002025-07-17T11:34:50.072ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = sled
48012025-07-17T11:34:50.073ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 1 sp_type = sled
48022025-07-17T11:34:50.073ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 1 sp_type = sled
48032025-07-17T11:34:50.073ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = sled
48042025-07-17T11:34:50.073ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 1 sp_type = sled
48052025-07-17T11:34:50.075ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 0 sp_type = sled start_ena = None
48062025-07-17T11:34:50.075ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
48072025-07-17T11:34:50.075ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 310.923149ms iteration = 19 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
48082025-07-17T11:34:50.076ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd401d76-6691-4ce8-9414-1471b689c956", "content-length": "81", "date": "Thu, 17 Jul 2025 11:34:49 GMT"} })
48092025-07-17T11:34:50.081ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
48102025-07-17T11:34:50.082ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
48112025-07-17T11:34:50.084ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48122025-07-17T11:34:50.088ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
48132025-07-17T11:34:50.088ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
48142025-07-17T11:34:50.088ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
48152025-07-17T11:34:50.091ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48162025-07-17T11:34:50.094ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
48172025-07-17T11:34:50.094ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
48182025-07-17T11:34:50.094ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
48192025-07-17T11:34:50.101ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1194 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 zone_address = fd00:1122:3344:101::2
48202025-07-17T11:34:50.101ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
48212025-07-17T11:34:50.101ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b12d6ff-5f6e-4ade-980f-686cff627429", "content-length": "26", "date": "Thu, 17 Jul 2025 11:34:49 GMT"} })
48222025-07-17T11:34:50.101ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1197 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
48232025-07-17T11:34:50.106ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
48242025-07-17T11:34:50.123ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
48252025-07-17T11:34:50.123ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
48262025-07-17T11:34:50.127ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 0 sp_type = switch start_ena = None
48272025-07-17T11:34:50.127ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
48282025-07-17T11:34:50.127ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "e3ff59d1-bf53-4e25-aa5f-e2841c7bf321", "content-length": "81", "date": "Thu, 17 Jul 2025 11:34:49 GMT"} })
48292025-07-17T11:34:50.129ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 451.869656ms iteration = 19 status = null
48302025-07-17T11:34:50.138ZINFOnexus (BackgroundTasks): VPC route manager sled 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
48312025-07-17T11:34:50.138ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
48322025-07-17T11:34:50.138ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "27f9c93d-d256-4fe2-9a3d-7e541c9f0a2a", "content-length": "372", "date": "Thu, 17 Jul 2025 11:34:50 GMT"} })
48332025-07-17T11:34:50.138ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
48342025-07-17T11:34:50.139ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
48352025-07-17T11:34:50.139ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 1 sp_type = sled start_ena = None
48362025-07-17T11:34:50.139ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/1/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
48372025-07-17T11:34:50.140ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/1/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4da91a4-2bdc-4982-a686-7800e2e13c22", "content-length": "81", "date": "Thu, 17 Jul 2025 11:34:49 GMT"} })
48382025-07-17T11:34:50.142ZDEBGnexus (BackgroundTasks): activation complete background_task = sp_ereport_ingester elapsed = 377.556537ms iteration = 19 status = {"errors":[],"sps":[]}
48392025-07-17T11:34:50.157ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
48402025-07-17T11:34:50.157ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
48412025-07-17T11:34:50.157ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "97aea07d-d8eb-4b3c-b8c9-4711e48d7a26", "date": "Thu, 17 Jul 2025 11:34:50 GMT"} })
48422025-07-17T11:34:50.157ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
48432025-07-17T11:34:50.158ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
48442025-07-17T11:34:50.160ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
48452025-07-17T11:34:50.161ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
48462025-07-17T11:34:50.165ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
48472025-07-17T11:34:50.165ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
48482025-07-17T11:34:50.168ZINFOnexus (BackgroundTasks): VPC route manager sled 0ae78f24-48db-4cf7-91cb-8d5311f2a17f push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
48492025-07-17T11:34:50.168ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
48502025-07-17T11:34:50.168ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "f4050330-ca87-431a-bee0-6c9af154f481", "date": "Thu, 17 Jul 2025 11:34:50 GMT"} })
48512025-07-17T11:34:50.171ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 406.961324ms iteration = 19 status = {}
48522025-07-17T11:34:50.185ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
48532025-07-17T11:34:50.186ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 451.915057ms iteration = 19 status = {}
48542025-07-17T11:34:50.195ZINFOnexus (BackgroundTasks): instance reincarnation completed successfully background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:113 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0
48552025-07-17T11:34:50.195ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 509.0318ms iteration = 10 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
48562025-07-17T11:34:50.218ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
48572025-07-17T11:34:50.218ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b45194f-acf9-4743-b64d-c30d86a82597", "content-length": "691", "date": "Thu, 17 Jul 2025 11:34:50 GMT"} })
48582025-07-17T11:34:50.218ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
48592025-07-17T11:34:50.218ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
48602025-07-17T11:34:50.218ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
48612025-07-17T11:34:50.221ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 456.451087ms iteration = 19 status = {}
48622025-07-17T11:34:50.249ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
48632025-07-17T11:34:50.250ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "80248917-10cf-44a0-95f2-b70010dd3a5f", "date": "Thu, 17 Jul 2025 11:34:50 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
48642025-07-17T11:34:50.250ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
48652025-07-17T11:34:50.251ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "aed48ae4-2ad0-4c09-b8da-dc02e1d73ff5", "date": "Thu, 17 Jul 2025 11:34:50 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
48662025-07-17T11:34:50.251ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
48672025-07-17T11:34:50.254ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "b4d53fa0-4b20-4cab-b36b-1617cc4c2f6a", "date": "Thu, 17 Jul 2025 11:34:50 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
48682025-07-17T11:34:50.254ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
48692025-07-17T11:34:50.256ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
48702025-07-17T11:34:50.256ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 578.904031ms iteration = 12 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
48712025-07-17T11:34:50.302ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
48722025-07-17T11:34:50.302ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
48732025-07-17T11:34:50.304ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 151caa7c-6a05-4995-94b4-7946d428117d, rack_id: 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5, switch_location: "switch0", port_name: Name(Name("qsfp0")), port_settings_id: Some(2e30c7ae-26da-46e4-9127-aac194921f20) } rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0
48742025-07-17T11:34:50.304ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
48752025-07-17T11:34:50.305ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
48762025-07-17T11:34:50.315ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1832 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = Name(Name("qsfp0"))
48772025-07-17T11:34:50.315ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1859 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0 switch_port_id = Name(Name("qsfp0")) switch_port_settings_id = 2e30c7ae-26da-46e4-9127-aac194921f20
48782025-07-17T11:34:50.316ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
48792025-07-17T11:34:50.316ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
48802025-07-17T11:34:50.316ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
48812025-07-17T11:34:50.316ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
48822025-07-17T11:34:50.316ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
48832025-07-17T11:34:50.316ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2006 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
48842025-07-17T11:34:50.316ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
48852025-07-17T11:34:50.316ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2042 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
48862025-07-17T11:34:50.316ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
48872025-07-17T11:34:50.316ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
48882025-07-17T11:34:50.317ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
48892025-07-17T11:34:50.319ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0
48902025-07-17T11:34:50.319ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
48912025-07-17T11:34:50.440ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "dcd6a9d4-d6e6-49ca-8c47-b0beb96dc5d8", "date": "Thu, 17 Jul 2025 11:34:50 GMT"} })
48922025-07-17T11:34:50.440ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch = Switch0
48932025-07-17T11:34:50.440ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
48942025-07-17T11:34:50.441ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "c08b5732-6508-4103-87b3-4e4459955c61", "content-length": "682", "date": "Thu, 17 Jul 2025 11:34:50 GMT"} })
48952025-07-17T11:34:50.441ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
48962025-07-17T11:34:50.441ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
48972025-07-17T11:34:50.445ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
48982025-07-17T11:34:50.445ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
48992025-07-17T11:34:50.448ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
49002025-07-17T11:34:50.448ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
49012025-07-17T11:34:50.450ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
49022025-07-17T11:34:50.450ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
49032025-07-17T11:34:50.456ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 691.898891ms iteration = 19 status = {}
49042025-07-17T11:34:51.692ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 463 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = ff3b72a4-79a9-4b78-a323-568ddcc719ad response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
49052025-07-17T11:34:54.764ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 110 reason = Timeout
49062025-07-17T11:34:54.765ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
49072025-07-17T11:34:54.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
49082025-07-17T11:34:54.766ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49092025-07-17T11:34:54.769ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
49102025-07-17T11:34:54.769ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
49112025-07-17T11:34:54.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
49122025-07-17T11:34:54.771ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
49132025-07-17T11:34:54.771ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 6.347434ms iteration = 110 status = {"chicken_switches_updated":false}
49142025-07-17T11:34:59.764ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 56 reason = Timeout
49152025-07-17T11:34:59.764ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 56 reason = Timeout
49162025-07-17T11:34:59.764ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
49172025-07-17T11:34:59.764ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
49182025-07-17T11:34:59.764ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 111 reason = Timeout
49192025-07-17T11:34:59.764ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
49202025-07-17T11:34:59.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
49212025-07-17T11:34:59.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
49222025-07-17T11:34:59.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
49232025-07-17T11:34:59.766ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49242025-07-17T11:34:59.767ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
49252025-07-17T11:34:59.767ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49262025-07-17T11:34:59.767ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
49272025-07-17T11:34:59.769ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.747563ms iteration = 56 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
49282025-07-17T11:34:59.770ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
49292025-07-17T11:34:59.770ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
49302025-07-17T11:34:59.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
49312025-07-17T11:34:59.770ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
49322025-07-17T11:34:59.770ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
49332025-07-17T11:34:59.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
49342025-07-17T11:34:59.771ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
49352025-07-17T11:34:59.771ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 7.304436ms iteration = 111 status = {"chicken_switches_updated":false}
49362025-07-17T11:34:59.772ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
49372025-07-17T11:34:59.772ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
49382025-07-17T11:34:59.773ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49392025-07-17T11:34:59.776ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
49402025-07-17T11:34:59.776ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
49412025-07-17T11:34:59.776ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
49422025-07-17T11:34:59.788ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
49432025-07-17T11:34:59.788ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 23.847104ms iteration = 56 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
49442025-07-17T11:35:01.684ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:50893 req_id = 663e65dc-1efe-4f32-8d57-963c5caa095b roles = RoleSet { roles: {} } uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?sort_by=id_ascending
49452025-07-17T11:35:01.684ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:50893 req_id = 663e65dc-1efe-4f32-8d57-963c5caa095b resource = Database result = Ok(()) uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?sort_by=id_ascending
49462025-07-17T11:35:01.687ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 2776 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:50893 req_id = 663e65dc-1efe-4f32-8d57-963c5caa095b response_code = 200 uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?sort_by=id_ascending
49472025-07-17T11:35:01.687ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:50893 req_id = 607add1c-cd12-4c45-87ff-80c614009e6b roles = RoleSet { roles: {} } uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmMzlkM2U3MC02NmU0LTQ2OGEtYTQ0ZC03NDZkMjJiZWEzYTAifX0%3D
49482025-07-17T11:35:01.687ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:50893 req_id = 607add1c-cd12-4c45-87ff-80c614009e6b resource = Database result = Ok(()) uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmMzlkM2U3MC02NmU0LTQ2OGEtYTQ0ZC03NDZkMjJiZWEzYTAifX0%3D
49492025-07-17T11:35:01.689ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 2501 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:50893 req_id = 607add1c-cd12-4c45-87ff-80c614009e6b response_code = 200 uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmMzlkM2U3MC02NmU0LTQ2OGEtYTQ0ZC03NDZkMjJiZWEzYTAifX0%3D
49502025-07-17T11:35:01.693ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 383 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = e53eecbd-2356-40ad-9324-e2e7722a1016 response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
49512025-07-17T11:35:04.764ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 112 reason = Timeout
49522025-07-17T11:35:04.764ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
49532025-07-17T11:35:04.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
49542025-07-17T11:35:04.765ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49552025-07-17T11:35:04.768ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
49562025-07-17T11:35:04.768ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
49572025-07-17T11:35:04.768ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
49582025-07-17T11:35:04.770ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
49592025-07-17T11:35:04.770ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 6.184952ms iteration = 112 status = {"chicken_switches_updated":false}
49602025-07-17T11:35:09.764ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 57 reason = Timeout
49612025-07-17T11:35:09.764ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 57 reason = Timeout
49622025-07-17T11:35:09.764ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
49632025-07-17T11:35:09.764ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
49642025-07-17T11:35:09.764ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 113 reason = Timeout
49652025-07-17T11:35:09.764ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
49662025-07-17T11:35:09.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
49672025-07-17T11:35:09.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
49682025-07-17T11:35:09.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
49692025-07-17T11:35:09.767ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49702025-07-17T11:35:09.767ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
49712025-07-17T11:35:09.768ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
49722025-07-17T11:35:09.768ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49732025-07-17T11:35:09.769ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 5.078577ms iteration = 57 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
49742025-07-17T11:35:09.770ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
49752025-07-17T11:35:09.770ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
49762025-07-17T11:35:09.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
49772025-07-17T11:35:09.771ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
49782025-07-17T11:35:09.771ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
49792025-07-17T11:35:09.772ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
49802025-07-17T11:35:09.773ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
49812025-07-17T11:35:09.773ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
49822025-07-17T11:35:09.773ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
49832025-07-17T11:35:09.773ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 9.076349ms iteration = 113 status = {"chicken_switches_updated":false}
49842025-07-17T11:35:09.774ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49852025-07-17T11:35:09.777ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
49862025-07-17T11:35:09.777ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
49872025-07-17T11:35:09.777ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
49882025-07-17T11:35:09.787ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
49892025-07-17T11:35:09.787ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.987963ms iteration = 57 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
49902025-07-17T11:35:11.693ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 645 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = 64425e54-1b2c-49df-948c-147460540c62 response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
49912025-07-17T11:35:14.765ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 114 reason = Timeout
49922025-07-17T11:35:14.765ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
49932025-07-17T11:35:14.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
49942025-07-17T11:35:14.766ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49952025-07-17T11:35:14.769ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
49962025-07-17T11:35:14.769ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
49972025-07-17T11:35:14.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
49982025-07-17T11:35:14.771ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
49992025-07-17T11:35:14.771ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 6.275132ms iteration = 114 status = {"chicken_switches_updated":false}
50002025-07-17T11:35:19.677ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 20 reason = Timeout
50012025-07-17T11:35:19.677ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
50022025-07-17T11:35:19.677ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
50032025-07-17T11:35:19.679ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3913, tv_nsec: 430815610 } })
50042025-07-17T11:35:19.679ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1186
50052025-07-17T11:35:19.709ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1194 zone_address = fd00:1122:3344:101::2
50062025-07-17T11:35:19.709ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
50072025-07-17T11:35:19.710ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "5cf5ee37-2431-45c0-a796-ee531f9b01b3", "content-length": "26", "date": "Thu, 17 Jul 2025 11:35:19 GMT"} })
50082025-07-17T11:35:19.710ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1197 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
50092025-07-17T11:35:19.710ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
50102025-07-17T11:35:19.734ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 20 reason = Timeout
50112025-07-17T11:35:19.734ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 20 reason = Timeout
50122025-07-17T11:35:19.734ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
50132025-07-17T11:35:19.734ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
50142025-07-17T11:35:19.736ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3913, tv_nsec: 488432700 } })
50152025-07-17T11:35:19.737ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1186
50162025-07-17T11:35:19.737ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.352624ms iteration = 20 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
50172025-07-17T11:35:19.744ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
50182025-07-17T11:35:19.744ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
50192025-07-17T11:35:19.746ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 69.435995ms iteration = 20 status = null
50202025-07-17T11:35:19.764ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 20 reason = Timeout
50212025-07-17T11:35:19.764ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 20 reason = Timeout
50222025-07-17T11:35:19.764ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
50232025-07-17T11:35:19.764ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
50242025-07-17T11:35:19.764ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
50252025-07-17T11:35:19.764ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 20 reason = Timeout
50262025-07-17T11:35:19.764ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
50272025-07-17T11:35:19.764ZDEBGnexus (BackgroundTasks): activating background_task = sp_ereport_ingester iteration = 20 reason = Timeout
50282025-07-17T11:35:19.764ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 58 reason = Timeout
50292025-07-17T11:35:19.764ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
50302025-07-17T11:35:19.764ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 20 reason = Timeout
50312025-07-17T11:35:19.764ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
50322025-07-17T11:35:19.764ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
50332025-07-17T11:35:19.764ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 20 reason = Timeout
50342025-07-17T11:35:19.764ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 20 reason = Timeout
50352025-07-17T11:35:19.764ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
50362025-07-17T11:35:19.764ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
50372025-07-17T11:35:19.764ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 20 reason = Timeout
50382025-07-17T11:35:19.764ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 58 reason = Timeout
50392025-07-17T11:35:19.764ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
50402025-07-17T11:35:19.764ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 20 reason = Timeout
50412025-07-17T11:35:19.764ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
50422025-07-17T11:35:19.764ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
50432025-07-17T11:35:19.764ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 115 reason = Timeout
50442025-07-17T11:35:19.764ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
50452025-07-17T11:35:19.764ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 20 reason = Timeout
50462025-07-17T11:35:19.764ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
50472025-07-17T11:35:19.764ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 20 reason = Timeout
50482025-07-17T11:35:19.764ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 20 reason = Timeout
50492025-07-17T11:35:19.764ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
50502025-07-17T11:35:19.764ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
50512025-07-17T11:35:19.765ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 20 reason = Timeout
50522025-07-17T11:35:19.765ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 20 reason = Timeout
50532025-07-17T11:35:19.765ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
50542025-07-17T11:35:19.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
50552025-07-17T11:35:19.765ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
50562025-07-17T11:35:19.765ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 21 reason = Timeout
50572025-07-17T11:35:19.765ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
50582025-07-17T11:35:19.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
50592025-07-17T11:35:19.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
50602025-07-17T11:35:19.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
50612025-07-17T11:35:19.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
50622025-07-17T11:35:19.765ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12225, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3913, tv_nsec: 516794754 } })
50632025-07-17T11:35:19.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
50642025-07-17T11:35:19.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
50652025-07-17T11:35:19.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
50662025-07-17T11:35:19.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
50672025-07-17T11:35:19.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
50682025-07-17T11:35:19.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
50692025-07-17T11:35:19.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
50702025-07-17T11:35:19.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
50712025-07-17T11:35:19.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
50722025-07-17T11:35:19.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
50732025-07-17T11:35:19.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
50742025-07-17T11:35:19.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
50752025-07-17T11:35:19.767ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50762025-07-17T11:35:19.767ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50772025-07-17T11:35:19.767ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
50782025-07-17T11:35:19.768ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
50792025-07-17T11:35:19.769ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50802025-07-17T11:35:19.769ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50812025-07-17T11:35:19.769ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
50822025-07-17T11:35:19.769ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
50832025-07-17T11:35:19.769ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50842025-07-17T11:35:19.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
50852025-07-17T11:35:19.770ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50862025-07-17T11:35:19.770ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
50872025-07-17T11:35:19.770ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
50882025-07-17T11:35:19.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
50892025-07-17T11:35:19.771ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50902025-07-17T11:35:19.772ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50912025-07-17T11:35:19.772ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1194 zone_address = fd00:1122:3344:101::2
50922025-07-17T11:35:19.772ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
50932025-07-17T11:35:19.772ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
50942025-07-17T11:35:19.772ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "7124043c-b773-4f27-bc2c-28f2158c22ea", "content-length": "26", "date": "Thu, 17 Jul 2025 11:35:19 GMT"} })
50952025-07-17T11:35:19.772ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1197 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
50962025-07-17T11:35:19.772ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
50972025-07-17T11:35:19.773ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
50982025-07-17T11:35:19.773ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
50992025-07-17T11:35:19.773ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 9.1302ms iteration = 58 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
51002025-07-17T11:35:19.773ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
51012025-07-17T11:35:19.774ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
51022025-07-17T11:35:19.774ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
51032025-07-17T11:35:19.774ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
51042025-07-17T11:35:19.774ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
51052025-07-17T11:35:19.774ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
51062025-07-17T11:35:19.774ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
51072025-07-17T11:35:19.774ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
51082025-07-17T11:35:19.774ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51092025-07-17T11:35:19.775ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
51102025-07-17T11:35:19.775ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
51112025-07-17T11:35:19.776ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
51122025-07-17T11:35:19.776ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
51132025-07-17T11:35:19.776ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
51142025-07-17T11:35:19.776ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
51152025-07-17T11:35:19.776ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
51162025-07-17T11:35:19.776ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
51172025-07-17T11:35:19.776ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
51182025-07-17T11:35:19.776ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
51192025-07-17T11:35:19.776ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
51202025-07-17T11:35:19.776ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
51212025-07-17T11:35:19.776ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
51222025-07-17T11:35:19.776ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
51232025-07-17T11:35:19.776ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
51242025-07-17T11:35:19.777ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3913, tv_nsec: 528884793 } })
51252025-07-17T11:35:19.777ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
51262025-07-17T11:35:19.777ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1186 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
51272025-07-17T11:35:19.777ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
51282025-07-17T11:35:19.778ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
51292025-07-17T11:35:19.778ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
51302025-07-17T11:35:19.778ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
51312025-07-17T11:35:19.778ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 13.425207ms iteration = 115 status = {"chicken_switches_updated":false}
51322025-07-17T11:35:19.778ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
51332025-07-17T11:35:19.778ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
51342025-07-17T11:35:19.778ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
51352025-07-17T11:35:19.779ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:550 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
51362025-07-17T11:35:19.779ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
51372025-07-17T11:35:19.779ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 14.564312ms iteration = 20 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
51382025-07-17T11:35:19.779ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
51392025-07-17T11:35:19.779ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51402025-07-17T11:35:19.780ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51412025-07-17T11:35:19.781ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 16.065082ms iteration = 20 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
51422025-07-17T11:35:19.783ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
51432025-07-17T11:35:19.783ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
51442025-07-17T11:35:19.783ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
51452025-07-17T11:35:19.784ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
51462025-07-17T11:35:19.784ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
51472025-07-17T11:35:19.784ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
51482025-07-17T11:35:19.808ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
51492025-07-17T11:35:19.808ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 43.941379ms iteration = 58 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
51502025-07-17T11:35:19.832ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
51512025-07-17T11:35:19.832ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
51522025-07-17T11:35:19.832ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
51532025-07-17T11:35:19.832ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
51542025-07-17T11:35:19.835ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
51552025-07-17T11:35:19.835ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
51562025-07-17T11:35:19.835ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
51572025-07-17T11:35:19.835ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
51582025-07-17T11:35:19.837ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
51592025-07-17T11:35:19.838ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
51602025-07-17T11:35:19.843ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
51612025-07-17T11:35:19.844ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
51622025-07-17T11:35:19.845ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 80.642143ms iteration = 20 status = {"errors":[],"finish_invoked_ok":[]}
51632025-07-17T11:35:19.845ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 80.854106ms iteration = 20 status = {"errors":[],"garbage_collect_requested":[]}
51642025-07-17T11:35:19.846ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 81.540415ms iteration = 20 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
51652025-07-17T11:35:19.853ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
51662025-07-17T11:35:19.853ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
51672025-07-17T11:35:19.853ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
51682025-07-17T11:35:19.853ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 89.107895ms iteration = 20 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
51692025-07-17T11:35:19.854ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
51702025-07-17T11:35:19.854ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/all-sp-ids
51712025-07-17T11:35:19.855ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/all-sp-ids", status: 200, headers: {"content-type": "application/json", "x-request-id": "e2d22f47-2ce4-4c60-9d8b-bf7d3c124a02", "content-length": "78", "date": "Thu, 17 Jul 2025 11:35:19 GMT"} })
51722025-07-17T11:35:19.855ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = sled
51732025-07-17T11:35:19.855ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = switch
51742025-07-17T11:35:19.855ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 1 sp_type = sled
51752025-07-17T11:35:19.855ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = sled
51762025-07-17T11:35:19.855ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 1 sp_type = sled
51772025-07-17T11:35:19.855ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = switch
51782025-07-17T11:35:19.865ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51792025-07-17T11:35:19.868ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 104.123443ms iteration = 20 status = {"errors":[],"requests_created_ok":[]}
51802025-07-17T11:35:19.870ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 0ae78f24-48db-4cf7-91cb-8d5311f2a17f, lookup_type: ById(0ae78f24-48db-4cf7-91cb-8d5311f2a17f) } result = Ok(())
51812025-07-17T11:35:19.870ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
51822025-07-17T11:35:19.870ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 1 sp_type = sled
51832025-07-17T11:35:19.871ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
51842025-07-17T11:35:19.871ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 0 sp_type = switch
51852025-07-17T11:35:19.871ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 0 sp_type = sled
51862025-07-17T11:35:19.872ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 107.068721ms iteration = 20 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
51872025-07-17T11:35:19.872ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
51882025-07-17T11:35:19.873ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
51892025-07-17T11:35:19.874ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 109.21855ms iteration = 20 status = 0
51902025-07-17T11:35:19.875ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 0 sp_type = sled
51912025-07-17T11:35:19.875ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = sled
51922025-07-17T11:35:19.875ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = sled
51932025-07-17T11:35:19.875ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 1 sp_type = sled
51942025-07-17T11:35:19.875ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 1 sp_type = sled
51952025-07-17T11:35:19.875ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 0 sp_type = switch
51962025-07-17T11:35:19.875ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = switch
51972025-07-17T11:35:19.875ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 1 sp_type = sled
51982025-07-17T11:35:19.876ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = switch
51992025-07-17T11:35:19.889ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:372
52002025-07-17T11:35:19.890ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 125.056028ms iteration = 20 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
52012025-07-17T11:35:19.892ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52022025-07-17T11:35:19.896ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
52032025-07-17T11:35:19.896ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
52042025-07-17T11:35:19.896ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
52052025-07-17T11:35:19.898ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 164.196865ms iteration = 20 status = {}
52062025-07-17T11:35:19.907ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1194 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 zone_address = fd00:1122:3344:101::2
52072025-07-17T11:35:19.907ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
52082025-07-17T11:35:19.907ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "1be892f7-6355-45a3-897c-83e51c161d78", "content-length": "26", "date": "Thu, 17 Jul 2025 11:35:19 GMT"} })
52092025-07-17T11:35:19.907ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1197 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
52102025-07-17T11:35:19.910ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
52112025-07-17T11:35:19.959ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 0 sp_type = sled start_ena = None
52122025-07-17T11:35:19.959ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
52132025-07-17T11:35:19.960ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "bca2416b-a0cf-47ac-9716-88beea140544", "content-length": "81", "date": "Thu, 17 Jul 2025 11:35:19 GMT"} })
52142025-07-17T11:35:19.969ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 1 sp_type = sled start_ena = None
52152025-07-17T11:35:19.969ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/1/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
52162025-07-17T11:35:19.970ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/1/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "47cc20b0-fa36-4fa4-9962-10fef6e15be7", "content-length": "81", "date": "Thu, 17 Jul 2025 11:35:19 GMT"} })
52172025-07-17T11:35:19.970ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 0 sp_type = switch start_ena = None
52182025-07-17T11:35:19.971ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
52192025-07-17T11:35:19.971ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "2116d09f-deaf-4633-9e19-77cfa2155ef2", "content-length": "81", "date": "Thu, 17 Jul 2025 11:35:19 GMT"} })
52202025-07-17T11:35:19.973ZDEBGnexus (BackgroundTasks): activation complete background_task = sp_ereport_ingester elapsed = 209.177298ms iteration = 20 status = {"errors":[],"sps":[]}
52212025-07-17T11:35:19.974ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 209.424751ms iteration = 21 status = {"physical_disks_added":0}
52222025-07-17T11:35:20.023ZINFOnexus (BackgroundTasks): VPC route manager sled 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
52232025-07-17T11:35:20.023ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
52242025-07-17T11:35:20.024ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f3c7b5f-2146-46eb-bf2e-b04e409324a0", "content-length": "372", "date": "Thu, 17 Jul 2025 11:35:19 GMT"} })
52252025-07-17T11:35:20.024ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
52262025-07-17T11:35:20.025ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
52272025-07-17T11:35:20.064ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
52282025-07-17T11:35:20.064ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
52292025-07-17T11:35:20.065ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
52302025-07-17T11:35:20.066ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
52312025-07-17T11:35:20.066ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 151caa7c-6a05-4995-94b4-7946d428117d, rack_id: 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5, switch_location: "switch0", port_name: Name(Name("qsfp0")), port_settings_id: Some(2e30c7ae-26da-46e4-9127-aac194921f20) } rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0
52322025-07-17T11:35:20.067ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
52332025-07-17T11:35:20.067ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
52342025-07-17T11:35:20.068ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52352025-07-17T11:35:20.070ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
52362025-07-17T11:35:20.070ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
52372025-07-17T11:35:20.070ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
52382025-07-17T11:35:20.072ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52392025-07-17T11:35:20.075ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
52402025-07-17T11:35:20.075ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
52412025-07-17T11:35:20.075ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
52422025-07-17T11:35:20.083ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1832 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = Name(Name("qsfp0"))
52432025-07-17T11:35:20.083ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1859 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0 switch_port_id = Name(Name("qsfp0")) switch_port_settings_id = 2e30c7ae-26da-46e4-9127-aac194921f20
52442025-07-17T11:35:20.084ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
52452025-07-17T11:35:20.084ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
52462025-07-17T11:35:20.084ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
52472025-07-17T11:35:20.084ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
52482025-07-17T11:35:20.084ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
52492025-07-17T11:35:20.084ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2006 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
52502025-07-17T11:35:20.084ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
52512025-07-17T11:35:20.084ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2042 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
52522025-07-17T11:35:20.084ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
52532025-07-17T11:35:20.084ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
52542025-07-17T11:35:20.085ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
52552025-07-17T11:35:20.088ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0
52562025-07-17T11:35:20.088ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
52572025-07-17T11:35:20.097ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
52582025-07-17T11:35:20.097ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
52592025-07-17T11:35:20.098ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "764f0271-1352-47b2-aaa2-6704324a6bb2", "date": "Thu, 17 Jul 2025 11:35:20 GMT"} })
52602025-07-17T11:35:20.098ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
52612025-07-17T11:35:20.099ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
52622025-07-17T11:35:20.102ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
52632025-07-17T11:35:20.102ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
52642025-07-17T11:35:20.106ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
52652025-07-17T11:35:20.106ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
52662025-07-17T11:35:20.110ZINFOnexus (BackgroundTasks): VPC route manager sled 0ae78f24-48db-4cf7-91cb-8d5311f2a17f push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
52672025-07-17T11:35:20.110ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
52682025-07-17T11:35:20.116ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
52692025-07-17T11:35:20.119ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "6f92f8db-cdf5-4cd6-bf8f-d2bda3eca383", "date": "Thu, 17 Jul 2025 11:35:20 GMT"} })
52702025-07-17T11:35:20.119ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "00677e63-6f94-49b5-9cbc-c28d1640ef17", "content-length": "691", "date": "Thu, 17 Jul 2025 11:35:20 GMT"} })
52712025-07-17T11:35:20.119ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
52722025-07-17T11:35:20.119ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
52732025-07-17T11:35:20.119ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
52742025-07-17T11:35:20.121ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 356.478949ms iteration = 20 status = {}
52752025-07-17T11:35:20.121ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 356.62559ms iteration = 20 status = {}
52762025-07-17T11:35:20.217ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "6fdec1ea-6e13-4247-af72-ec9bfe95b80a", "date": "Thu, 17 Jul 2025 11:35:19 GMT"} })
52772025-07-17T11:35:20.217ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch = Switch0
52782025-07-17T11:35:20.217ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
52792025-07-17T11:35:20.224ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a2d226d-0879-4493-9039-49327291d29f", "content-length": "682", "date": "Thu, 17 Jul 2025 11:35:20 GMT"} })
52802025-07-17T11:35:20.224ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
52812025-07-17T11:35:20.224ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
52822025-07-17T11:35:20.228ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
52832025-07-17T11:35:20.228ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
52842025-07-17T11:35:20.232ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
52852025-07-17T11:35:20.232ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
52862025-07-17T11:35:20.234ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
52872025-07-17T11:35:20.235ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
52882025-07-17T11:35:20.242ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 477.684817ms iteration = 20 status = {}
52892025-07-17T11:35:21.693ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 355 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = 50270cfb-4958-45bf-ace0-8eded847ff34 response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
52902025-07-17T11:35:24.765ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 116 reason = Timeout
52912025-07-17T11:35:24.765ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
52922025-07-17T11:35:24.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
52932025-07-17T11:35:24.767ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52942025-07-17T11:35:24.770ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
52952025-07-17T11:35:24.770ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
52962025-07-17T11:35:24.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
52972025-07-17T11:35:24.772ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
52982025-07-17T11:35:24.772ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 7.291686ms iteration = 116 status = {"chicken_switches_updated":false}
52992025-07-17T11:35:29.764ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 59 reason = Timeout
53002025-07-17T11:35:29.764ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
53012025-07-17T11:35:29.764ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 59 reason = Timeout
53022025-07-17T11:35:29.764ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
53032025-07-17T11:35:29.764ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 117 reason = Timeout
53042025-07-17T11:35:29.764ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
53052025-07-17T11:35:29.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
53062025-07-17T11:35:29.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
53072025-07-17T11:35:29.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
53082025-07-17T11:35:29.766ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
53092025-07-17T11:35:29.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
53102025-07-17T11:35:29.767ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53112025-07-17T11:35:29.767ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53122025-07-17T11:35:29.768ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.854091ms iteration = 59 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
53132025-07-17T11:35:29.770ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
53142025-07-17T11:35:29.770ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
53152025-07-17T11:35:29.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
53162025-07-17T11:35:29.770ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
53172025-07-17T11:35:29.770ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
53182025-07-17T11:35:29.771ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
53192025-07-17T11:35:29.772ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
53202025-07-17T11:35:29.772ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
53212025-07-17T11:35:29.772ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
53222025-07-17T11:35:29.772ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 8.429071ms iteration = 117 status = {"chicken_switches_updated":false}
53232025-07-17T11:35:29.774ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53242025-07-17T11:35:29.777ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
53252025-07-17T11:35:29.777ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
53262025-07-17T11:35:29.777ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
53272025-07-17T11:35:29.786ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
53282025-07-17T11:35:29.786ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 21.676116ms iteration = 59 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
53292025-07-17T11:35:31.685ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:50893 req_id = d505ddb1-af10-45b8-9978-01253d8ec900 roles = RoleSet { roles: {} } uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?sort_by=id_ascending
53302025-07-17T11:35:31.685ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:50893 req_id = d505ddb1-af10-45b8-9978-01253d8ec900 resource = Database result = Ok(()) uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?sort_by=id_ascending
53312025-07-17T11:35:31.688ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 3031 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:50893 req_id = d505ddb1-af10-45b8-9978-01253d8ec900 response_code = 200 uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?sort_by=id_ascending
53322025-07-17T11:35:31.688ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:50893 req_id = b7c64616-1f17-41c2-adb9-84de47b5d1ba roles = RoleSet { roles: {} } uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmMzlkM2U3MC02NmU0LTQ2OGEtYTQ0ZC03NDZkMjJiZWEzYTAifX0%3D
53332025-07-17T11:35:31.688ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:50893 req_id = b7c64616-1f17-41c2-adb9-84de47b5d1ba resource = Database result = Ok(()) uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmMzlkM2U3MC02NmU0LTQ2OGEtYTQ0ZC03NDZkMjJiZWEzYTAifX0%3D
53342025-07-17T11:35:31.690ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 1913 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:50893 req_id = b7c64616-1f17-41c2-adb9-84de47b5d1ba response_code = 200 uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmMzlkM2U3MC02NmU0LTQ2OGEtYTQ0ZC03NDZkMjJiZWEzYTAifX0%3D
53352025-07-17T11:35:31.693ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 283 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = 7fbe5776-e74b-4204-8ef2-5a12f3cd0b3a response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
53362025-07-17T11:35:34.765ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 118 reason = Timeout
53372025-07-17T11:35:34.765ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
53382025-07-17T11:35:34.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
53392025-07-17T11:35:34.767ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53402025-07-17T11:35:34.770ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
53412025-07-17T11:35:34.770ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
53422025-07-17T11:35:34.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
53432025-07-17T11:35:34.772ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
53442025-07-17T11:35:34.772ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 7.031763ms iteration = 118 status = {"chicken_switches_updated":false}
53452025-07-17T11:35:39.765ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 60 reason = Timeout
53462025-07-17T11:35:39.765ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 60 reason = Timeout
53472025-07-17T11:35:39.765ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
53482025-07-17T11:35:39.765ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 119 reason = Timeout
53492025-07-17T11:35:39.765ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
53502025-07-17T11:35:39.765ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
53512025-07-17T11:35:39.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
53522025-07-17T11:35:39.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
53532025-07-17T11:35:39.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
53542025-07-17T11:35:39.767ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53552025-07-17T11:35:39.768ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53562025-07-17T11:35:39.768ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
53572025-07-17T11:35:39.768ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
53582025-07-17T11:35:39.770ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
53592025-07-17T11:35:39.770ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
53602025-07-17T11:35:39.770ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 5.004746ms iteration = 60 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
53612025-07-17T11:35:39.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
53622025-07-17T11:35:39.771ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
53632025-07-17T11:35:39.771ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
53642025-07-17T11:35:39.771ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
53652025-07-17T11:35:39.772ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
53662025-07-17T11:35:39.773ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
53672025-07-17T11:35:39.773ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
53682025-07-17T11:35:39.773ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 7.851874ms iteration = 119 status = {"chicken_switches_updated":false}
53692025-07-17T11:35:39.774ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53702025-07-17T11:35:39.777ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
53712025-07-17T11:35:39.777ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
53722025-07-17T11:35:39.777ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
53732025-07-17T11:35:39.791ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
53742025-07-17T11:35:39.791ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 26.069064ms iteration = 60 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
53752025-07-17T11:35:41.693ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 457 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = d4bf0eab-e472-4fc9-8a67-82bedfd09789 response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
53762025-07-17T11:35:44.764ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 120 reason = Timeout
53772025-07-17T11:35:44.764ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
53782025-07-17T11:35:44.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
53792025-07-17T11:35:44.766ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53802025-07-17T11:35:44.769ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
53812025-07-17T11:35:44.769ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
53822025-07-17T11:35:44.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
53832025-07-17T11:35:44.771ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
53842025-07-17T11:35:44.771ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 7.011473ms iteration = 120 status = {"chicken_switches_updated":false}
53852025-07-17T11:35:49.677ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 11 reason = Timeout
53862025-07-17T11:35:49.677ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 11 reason = Timeout
53872025-07-17T11:35:49.677ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 11 reason = Timeout
53882025-07-17T11:35:49.677ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
53892025-07-17T11:35:49.677ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
53902025-07-17T11:35:49.677ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
53912025-07-17T11:35:49.677ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
53922025-07-17T11:35:49.677ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 13 reason = Timeout
53932025-07-17T11:35:49.677ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 21 reason = Timeout
53942025-07-17T11:35:49.677ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 11 reason = Timeout
53952025-07-17T11:35:49.677ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-07-17 11:25:49.677113115 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
53962025-07-17T11:35:49.677ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
53972025-07-17T11:35:49.677ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 11 reason = Timeout
53982025-07-17T11:35:49.677ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
53992025-07-17T11:35:49.677ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
54002025-07-17T11:35:49.677ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 11 reason = Timeout
54012025-07-17T11:35:49.677ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 13 reason = Timeout
54022025-07-17T11:35:49.677ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
54032025-07-17T11:35:49.677ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
54042025-07-17T11:35:49.677ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
54052025-07-17T11:35:49.677ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
54062025-07-17T11:35:49.677ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("3dd3bf3b-7760-4750-b1d3-52ee777335f0.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("79e5c77d-18b3-4951-8d11-cfa3eae2b3e3.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("79e5c77d-18b3-4951-8d11-cfa3eae2b3e3.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::8)) }], valid_until: Instant { tv_sec: 3943, tv_nsec: 429223366 } })
54072025-07-17T11:35:49.677ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
54082025-07-17T11:35:49.677ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("3144a75a-317f-4b68-93e2-a210ca1189d1.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("87408888-9df4-4e0c-bc8e-2f920d039100.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("ee4378c5-e5f2-41cf-8969-e5e03e1b9a41.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("ee4378c5-e5f2-41cf-8969-e5e03e1b9a41.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:3::1)) }], valid_until: Instant { tv_sec: 3943, tv_nsec: 429323688 } })
54092025-07-17T11:35:49.678ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
54102025-07-17T11:35:49.678ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.205216ms iteration = 11 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
54112025-07-17T11:35:49.678ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } dns_group = internal
54122025-07-17T11:35:49.678ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.235176ms iteration = 11 status = {"addresses":["[fd00:1122:3344:1::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:3::1]:5353"]}
54132025-07-17T11:35:49.678ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
54142025-07-17T11:35:49.679ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
54152025-07-17T11:35:49.679ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
54162025-07-17T11:35:49.679ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
54172025-07-17T11:35:49.680ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-07-17 11:25:49.677113115 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
54182025-07-17T11:35:49.680ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 3.076801ms iteration = 11 status = {"expiration":"2025-07-17T11:25:49.677113115Z","pruned":[]}
54192025-07-17T11:35:49.680ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 12 reason = Timeout
54202025-07-17T11:35:49.680ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:85 reason = blueprint disabled target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a
54212025-07-17T11:35:49.680ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 16.85\u{b5}s iteration = 12 status = {"enabled":false,"target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a"}
54222025-07-17T11:35:49.681ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54232025-07-17T11:35:49.683ZDEBGnexus (BackgroundTasks): activating background_task = inventory_collection iteration = 2 reason = Timeout
54242025-07-17T11:35:49.683ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
54252025-07-17T11:35:49.684ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
54262025-07-17T11:35:49.685ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
54272025-07-17T11:35:49.685ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
54282025-07-17T11:35:49.685ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 4 reason = Timeout
54292025-07-17T11:35:49.685ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
54302025-07-17T11:35:49.685ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_planner iteration = 13 reason = Timeout
54312025-07-17T11:35:49.685ZDEBGnexus (BackgroundTasks): blueprint planning disabled, doing nothing background_task = blueprint_planner
54322025-07-17T11:35:49.685ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_planner elapsed = 14.92\u{b5}s iteration = 13 status = "Disabled"
54332025-07-17T11:35:49.685ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 11 reason = Timeout
54342025-07-17T11:35:49.685ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 11 reason = Timeout
54352025-07-17T11:35:49.685ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:53
54362025-07-17T11:35:49.685ZDEBGnexus (BackgroundTasks): activating background_task = service_firewall_rule_propagation iteration = 3 reason = Timeout
54372025-07-17T11:35:49.685ZDEBGnexus (service-firewall-rule-progator): starting background task for service firewall rule propagation background_task = service_firewall_rule_propagation
54382025-07-17T11:35:49.685ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
54392025-07-17T11:35:49.685ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
54402025-07-17T11:35:49.685ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
54412025-07-17T11:35:49.685ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
54422025-07-17T11:35:49.686ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
54432025-07-17T11:35:49.686ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
54442025-07-17T11:35:49.686ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
54452025-07-17T11:35:49.686ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
54462025-07-17T11:35:49.686ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 11 reason = Timeout
54472025-07-17T11:35:49.686ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
54482025-07-17T11:35:49.688ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
54492025-07-17T11:35:49.688ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54502025-07-17T11:35:49.689ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54512025-07-17T11:35:49.691ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54522025-07-17T11:35:49.691ZDEBGnexus (BackgroundTasks): activating background_task = saga_recovery iteration = 2 reason = Timeout
54532025-07-17T11:35:49.691ZDEBGnexus (SagaRecoverer): listing candidate sagas for recovery
54542025-07-17T11:35:49.691ZDEBGnexus (SagaRecoverer): roles roles = RoleSet { roles: {} }
54552025-07-17T11:35:49.692ZDEBGnexus (SagaRecoverer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000003a8a, .. }) resource = Database result = Ok(())
54562025-07-17T11:35:49.692ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3943, tv_nsec: 443836729 } })
54572025-07-17T11:35:49.692ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
54582025-07-17T11:35:49.692ZDEBGnexus (BackgroundTasks): activating background_task = tuf_artifact_replication iteration = 3 reason = Timeout
54592025-07-17T11:35:49.692ZDEBGnexus (BackgroundTasks): activating background_task = webhook_deliverator iteration = 11 reason = Timeout
54602025-07-17T11:35:49.692ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
54612025-07-17T11:35:49.692ZDEBGnexus (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} }
54622025-07-17T11:35:49.692ZDEBGnexus (BackgroundTasks): activating background_task = alert_dispatcher iteration = 11 reason = Timeout
54632025-07-17T11:35:49.692ZDEBGnexus (BackgroundTasks): roles background_task = alert_dispatcher roles = RoleSet { roles: {} }
54642025-07-17T11:35:49.692ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 11 reason = Timeout
54652025-07-17T11:35:49.692ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
54662025-07-17T11:35:49.692ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
54672025-07-17T11:35:49.692ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1186
54682025-07-17T11:35:49.692ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
54692025-07-17T11:35:49.693ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
54702025-07-17T11:35:49.693ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database result = Ok(())
54712025-07-17T11:35:49.693ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = alert_dispatcher resource = Database result = Ok(())
54722025-07-17T11:35:49.693ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
54732025-07-17T11:35:49.695ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
54742025-07-17T11:35:49.695ZDEBGnexus (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
54752025-07-17T11:35:49.695ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
54762025-07-17T11:35:49.695ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
54772025-07-17T11:35:49.704ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54782025-07-17T11:35:49.704ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54792025-07-17T11:35:49.707ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
54802025-07-17T11:35:49.707ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
54812025-07-17T11:35:49.708ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
54822025-07-17T11:35:49.708ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
54832025-07-17T11:35:49.709ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
54842025-07-17T11:35:49.709ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Inventory result = Ok(())
54852025-07-17T11:35:49.709ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
54862025-07-17T11:35:49.710ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
54872025-07-17T11:35:49.715ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54882025-07-17T11:35:49.717ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 25.142082ms iteration = 11 status = {"errors":[],"found_port_ok":[]}
54892025-07-17T11:35:49.718ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54902025-07-17T11:35:49.721ZINFOnexus (BackgroundTasks): all webhook delivery tasks completed background_task = webhook_deliverator file = nexus/src/app/background/tasks/webhook_deliverator.rs:204 num_receivers = 0
54912025-07-17T11:35:49.721ZDEBGnexus (BackgroundTasks): activation complete background_task = webhook_deliverator elapsed = 28.453525ms iteration = 11 status = {"by_rx":{},"error":null}
54922025-07-17T11:35:49.722ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
54932025-07-17T11:35:49.722ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
54942025-07-17T11:35:49.723ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
54952025-07-17T11:35:49.723ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
54962025-07-17T11:35:49.723ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
54972025-07-17T11:35:49.723ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
54982025-07-17T11:35:49.723ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
54992025-07-17T11:35:49.723ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
55002025-07-17T11:35:49.724ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
55012025-07-17T11:35:49.724ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
55022025-07-17T11:35:49.727ZDEBGnexus (BackgroundTasks): roles background_task = alert_dispatcher roles = RoleSet { roles: {} }
55032025-07-17T11:35:49.728ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
55042025-07-17T11:35:49.728ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = alert_dispatcher resource = Database result = Ok(())
55052025-07-17T11:35:49.728ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
55062025-07-17T11:35:49.731ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
55072025-07-17T11:35:49.731ZDEBGnexus (BackgroundTasks): activation complete background_task = alert_dispatcher elapsed = 38.907993ms iteration = 11 status = {"dispatched":[],"errors":[],"glob_version":"161.0.0","globs_reprocessed":{},"no_receivers":[]}
55082025-07-17T11:35:49.731ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
55092025-07-17T11:35:49.734ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 21 reason = Timeout
55102025-07-17T11:35:49.734ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 21 reason = Timeout
55112025-07-17T11:35:49.734ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
55122025-07-17T11:35:49.734ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
55132025-07-17T11:35:49.734ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
55142025-07-17T11:35:49.734ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
55152025-07-17T11:35:49.736ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55162025-07-17T11:35:49.738ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3943, tv_nsec: 490294171 } })
55172025-07-17T11:35:49.739ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1186
55182025-07-17T11:35:49.739ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 5.803456ms iteration = 21 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
55192025-07-17T11:35:49.742ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
55202025-07-17T11:35:49.742ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
55212025-07-17T11:35:49.743ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
55222025-07-17T11:35:49.745ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
55232025-07-17T11:35:49.745ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
55242025-07-17T11:35:49.747ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
55252025-07-17T11:35:49.747ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
55262025-07-17T11:35:49.758ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
55272025-07-17T11:35:49.759ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
55282025-07-17T11:35:49.759ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
55292025-07-17T11:35:49.759ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
55302025-07-17T11:35:49.759ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
55312025-07-17T11:35:49.765ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 21 reason = Timeout
55322025-07-17T11:35:49.765ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
55332025-07-17T11:35:49.765ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 21 reason = Timeout
55342025-07-17T11:35:49.765ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
55352025-07-17T11:35:49.765ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 22 reason = Timeout
55362025-07-17T11:35:49.765ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 61 reason = Timeout
55372025-07-17T11:35:49.765ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
55382025-07-17T11:35:49.765ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
55392025-07-17T11:35:49.765ZDEBGnexus (BackgroundTasks): activating background_task = sp_ereport_ingester iteration = 21 reason = Timeout
55402025-07-17T11:35:49.765ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 21 reason = Timeout
55412025-07-17T11:35:49.765ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
55422025-07-17T11:35:49.765ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 61 reason = Timeout
55432025-07-17T11:35:49.765ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
55442025-07-17T11:35:49.765ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 21 reason = Timeout
55452025-07-17T11:35:49.765ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
55462025-07-17T11:35:49.766ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
55472025-07-17T11:35:49.766ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 121 reason = Timeout
55482025-07-17T11:35:49.766ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
55492025-07-17T11:35:49.766ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 21 reason = Timeout
55502025-07-17T11:35:49.766ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
55512025-07-17T11:35:49.766ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 21 reason = Timeout
55522025-07-17T11:35:49.766ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
55532025-07-17T11:35:49.766ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 21 reason = Timeout
55542025-07-17T11:35:49.766ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 21 reason = Timeout
55552025-07-17T11:35:49.766ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
55562025-07-17T11:35:49.766ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
55572025-07-17T11:35:49.766ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 21 reason = Timeout
55582025-07-17T11:35:49.766ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
55592025-07-17T11:35:49.766ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 21 reason = Timeout
55602025-07-17T11:35:49.766ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
55612025-07-17T11:35:49.766ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 21 reason = Timeout
55622025-07-17T11:35:49.766ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
55632025-07-17T11:35:49.766ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 21 reason = Timeout
55642025-07-17T11:35:49.766ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
55652025-07-17T11:35:49.766ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
55662025-07-17T11:35:49.766ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 21 reason = Timeout
55672025-07-17T11:35:49.766ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
55682025-07-17T11:35:49.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
55692025-07-17T11:35:49.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
55702025-07-17T11:35:49.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
55712025-07-17T11:35:49.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
55722025-07-17T11:35:49.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
55732025-07-17T11:35:49.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
55742025-07-17T11:35:49.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
55752025-07-17T11:35:49.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
55762025-07-17T11:35:49.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
55772025-07-17T11:35:49.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
55782025-07-17T11:35:49.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
55792025-07-17T11:35:49.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
55802025-07-17T11:35:49.766ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12225, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3943, tv_nsec: 518214409 } })
55812025-07-17T11:35:49.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
55822025-07-17T11:35:49.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
55832025-07-17T11:35:49.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
55842025-07-17T11:35:49.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
55852025-07-17T11:35:49.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
55862025-07-17T11:35:49.767ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55872025-07-17T11:35:49.769ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55882025-07-17T11:35:49.769ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55892025-07-17T11:35:49.773ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
55902025-07-17T11:35:49.773ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
55912025-07-17T11:35:49.773ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
55922025-07-17T11:35:49.774ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
55932025-07-17T11:35:49.774ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
55942025-07-17T11:35:49.774ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
55952025-07-17T11:35:49.774ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
55962025-07-17T11:35:49.774ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
55972025-07-17T11:35:49.774ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
55982025-07-17T11:35:49.774ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55992025-07-17T11:35:49.774ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
56002025-07-17T11:35:49.774ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
56012025-07-17T11:35:49.775ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56022025-07-17T11:35:49.776ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
56032025-07-17T11:35:49.777ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
56042025-07-17T11:35:49.778ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
56052025-07-17T11:35:49.778ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
56062025-07-17T11:35:49.778ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
56072025-07-17T11:35:49.778ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
56082025-07-17T11:35:49.779ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
56092025-07-17T11:35:49.779ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
56102025-07-17T11:35:49.779ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
56112025-07-17T11:35:49.779ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
56122025-07-17T11:35:49.780ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
56132025-07-17T11:35:49.780ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
56142025-07-17T11:35:49.781ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
56152025-07-17T11:35:49.781ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
56162025-07-17T11:35:49.781ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
56172025-07-17T11:35:49.781ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
56182025-07-17T11:35:49.781ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
56192025-07-17T11:35:49.782ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56202025-07-17T11:35:49.786ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
56212025-07-17T11:35:49.786ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
56222025-07-17T11:35:49.786ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
56232025-07-17T11:35:49.801ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56242025-07-17T11:35:49.802ZINFOnexus (SagaRecoverer): listed in-progress sagas count = 0 file = nexus/src/app/background/tasks/saga_recovery.rs:438
56252025-07-17T11:35:49.802ZDEBGnexus (BackgroundTasks): activation complete background_task = saga_recovery elapsed = 110.482386ms iteration = 2 status = {"last_pass":{"Success":{"nfailed":0,"nfound":0,"nrecovered":0,"nremoved":0,"nskipped":0}},"ntotal_failures":0,"ntotal_finished":0,"ntotal_recovered":0,"ntotal_sec_errors_bad_state":0,"ntotal_sec_errors_missing":0,"ntotal_started":0,"recent_failures":[],"recent_recoveries":[]}
56262025-07-17T11:35:49.804ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
56272025-07-17T11:35:49.805ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
56282025-07-17T11:35:49.805ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56292025-07-17T11:35:49.805ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
56302025-07-17T11:35:49.805ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
56312025-07-17T11:35:49.805ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
56322025-07-17T11:35:49.806ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3943, tv_nsec: 557829672 } })
56332025-07-17T11:35:49.806ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
56342025-07-17T11:35:49.806ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 915fbba0-947f-4bf0-9f9f-48a79df81f7b dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
56352025-07-17T11:35:49.806ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
56362025-07-17T11:35:49.806ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1186 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
56372025-07-17T11:35:49.806ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
56382025-07-17T11:35:49.806ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
56392025-07-17T11:35:49.806ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:138 found = 0 not_ready_to_be_deleted = 0
56402025-07-17T11:35:49.806ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 121.167518ms iteration = 11 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
56412025-07-17T11:35:49.807ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
56422025-07-17T11:35:49.807ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
56432025-07-17T11:35:49.808ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
56442025-07-17T11:35:49.808ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
56452025-07-17T11:35:49.809ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56462025-07-17T11:35:49.809ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56472025-07-17T11:35:49.810ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
56482025-07-17T11:35:49.810ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:170 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
56492025-07-17T11:35:49.810ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 123.250735ms iteration = 11 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
56502025-07-17T11:35:49.811ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
56512025-07-17T11:35:49.811ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
56522025-07-17T11:35:49.811ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
56532025-07-17T11:35:49.811ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
56542025-07-17T11:35:49.811ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
56552025-07-17T11:35:49.813ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
56562025-07-17T11:35:49.813ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
56572025-07-17T11:35:49.814ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
56582025-07-17T11:35:49.814ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
56592025-07-17T11:35:49.814ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
56602025-07-17T11:35:49.815ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
56612025-07-17T11:35:49.815ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
56622025-07-17T11:35:49.815ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56632025-07-17T11:35:49.815ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
56642025-07-17T11:35:49.818ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56652025-07-17T11:35:49.818ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56662025-07-17T11:35:49.818ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
56672025-07-17T11:35:49.819ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
56682025-07-17T11:35:49.821ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
56692025-07-17T11:35:49.821ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
56702025-07-17T11:35:49.822ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
56712025-07-17T11:35:49.822ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56722025-07-17T11:35:49.825ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
56732025-07-17T11:35:49.825ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
56742025-07-17T11:35:49.825ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
56752025-07-17T11:35:49.826ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
56762025-07-17T11:35:49.826ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
56772025-07-17T11:35:49.827ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
56782025-07-17T11:35:49.827ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
56792025-07-17T11:35:49.827ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
56802025-07-17T11:35:49.827ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:550 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
56812025-07-17T11:35:49.827ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 61.843255ms iteration = 21 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
56822025-07-17T11:35:49.827ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
56832025-07-17T11:35:49.830ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56842025-07-17T11:35:49.833ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
56852025-07-17T11:35:49.833ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
56862025-07-17T11:35:49.833ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
56872025-07-17T11:35:49.854ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 915fbba0-947f-4bf0-9f9f-48a79df81f7b dns_zone_name = control-plane.oxide.internal found_so_far = 59 version = 1
56882025-07-17T11:35:49.854ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
56892025-07-17T11:35:49.854ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-07-17 11:26:34.922237 UTC dns_group = internal generation = 1
56902025-07-17T11:35:49.854ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 177.227277ms iteration = 11 status = {"generation":1}
56912025-07-17T11:35:49.877ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 111.740504ms iteration = 21 status = {"errors":[],"finish_invoked_ok":[]}
56922025-07-17T11:35:49.880ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56932025-07-17T11:35:49.882ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56942025-07-17T11:35:49.887ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
56952025-07-17T11:35:49.887ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
56962025-07-17T11:35:49.887ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
56972025-07-17T11:35:49.887ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56982025-07-17T11:35:49.890ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
56992025-07-17T11:35:49.890ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
57002025-07-17T11:35:49.890ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
57012025-07-17T11:35:49.892ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
57022025-07-17T11:35:49.892ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
57032025-07-17T11:35:49.892ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
57042025-07-17T11:35:49.893ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
57052025-07-17T11:35:49.893ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
57062025-07-17T11:35:49.894ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
57072025-07-17T11:35:49.897ZDEBGnexus (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(45b28df7-4c98-4c20-9185-a692c52225c8, 0), (7fbb6201-cec7-4b05-9bc1-7dc1df30948d, 0), (bd15575d-5499-4736-988b-3d1df00d704b, 0)]
57082025-07-17T11:35:49.897ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12225, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3943, tv_nsec: 649273297 } })
57092025-07-17T11:35:49.907ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
57102025-07-17T11:35:49.907ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
57112025-07-17T11:35:49.913ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
57122025-07-17T11:35:49.913ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
57132025-07-17T11:35:49.916ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
57142025-07-17T11:35:49.916ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 150.597185ms iteration = 121 status = {"chicken_switches_updated":false}
57152025-07-17T11:35:49.917ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
57162025-07-17T11:35:49.917ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
57172025-07-17T11:35:49.920ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 154.888142ms iteration = 21 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
57182025-07-17T11:35:49.930ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
57192025-07-17T11:35:49.930ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
57202025-07-17T11:35:49.937ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 171.957667ms iteration = 61 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
57212025-07-17T11:35:49.938ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57222025-07-17T11:35:49.942ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
57232025-07-17T11:35:49.942ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
57242025-07-17T11:35:49.943ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
57252025-07-17T11:35:49.943ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
57262025-07-17T11:35:49.943ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
57272025-07-17T11:35:49.946ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
57282025-07-17T11:35:49.946ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
57292025-07-17T11:35:49.948ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 182.279663ms iteration = 21 status = {"errors":[],"requests_created_ok":[]}
57302025-07-17T11:35:49.949ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57312025-07-17T11:35:49.950ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
57322025-07-17T11:35:49.951ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
57332025-07-17T11:35:49.953ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
57342025-07-17T11:35:49.953ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
57352025-07-17T11:35:49.953ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
57362025-07-17T11:35:49.954ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 188.813579ms iteration = 21 status = {"errors":[],"garbage_collect_requested":[]}
57372025-07-17T11:35:49.955ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
57382025-07-17T11:35:49.955ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
57392025-07-17T11:35:49.956ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57402025-07-17T11:35:49.957ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
57412025-07-17T11:35:49.958ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
57422025-07-17T11:35:49.959ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
57432025-07-17T11:35:49.960ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
57442025-07-17T11:35:49.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
57452025-07-17T11:35:49.960ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
57462025-07-17T11:35:49.961ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
57472025-07-17T11:35:49.961ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
57482025-07-17T11:35:49.962ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
57492025-07-17T11:35:49.963ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57502025-07-17T11:35:49.963ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57512025-07-17T11:35:49.963ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
57522025-07-17T11:35:49.963ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
57532025-07-17T11:35:49.963ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
57542025-07-17T11:35:49.964ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
57552025-07-17T11:35:49.969ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
57562025-07-17T11:35:49.969ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
57572025-07-17T11:35:49.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
57582025-07-17T11:35:49.970ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
57592025-07-17T11:35:49.970ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
57602025-07-17T11:35:49.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
57612025-07-17T11:35:49.972ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-07-17T11:25:46.964149Z, time_modified: 2025-07-17T11:25:46.964149Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 1fa836d9-7854-4ada-8c89-c26d922646c6, db_silo: Silo { identity: SiloIdentity { id: 1fa836d9-7854-4ada-8c89-c26d922646c6, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-07-17T11:26:35.643175Z, time_modified: 2025-07-17T11:26:35.643175Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "d7c3455b70a66c86ec2a7e597ca46fe0ea0c49dde5c1363f198c20aa77d58442" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 1fa836d9-7854-4ada-8c89-c26d922646c6, db_silo: Silo { identity: SiloIdentity { id: 1fa836d9-7854-4ada-8c89-c26d922646c6, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-07-17T11:26:35.643175Z, time_modified: 2025-07-17T11:26:35.643175Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "d7c3455b70a66c86ec2a7e597ca46fe0ea0c49dde5c1363f198c20aa77d58442" }] }) }
57622025-07-17T11:35:49.972ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 295.784369ms iteration = 11 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"1fa836d9-7854-4ada-8c89-c26d922646c6","tls_certs":["d7c3455b70a66c86ec2a7e597ca46fe0ea0c49dde5c1363f198c20aa77d58442"]}},"default_endpoint":{"silo_id":"1fa836d9-7854-4ada-8c89-c26d922646c6","tls_certs":["d7c3455b70a66c86ec2a7e597ca46fe0ea0c49dde5c1363f198c20aa77d58442"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
57632025-07-17T11:35:49.975ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
57642025-07-17T11:35:49.975ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
57652025-07-17T11:35:49.978ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
57662025-07-17T11:35:49.978ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
57672025-07-17T11:35:49.980ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
57682025-07-17T11:35:49.980ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 4a9e4c5e-728e-4850-bd58-14548c02bf6f dns_zone_name = oxide.test found_so_far = 0 version = 2
57692025-07-17T11:35:49.980ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
57702025-07-17T11:35:49.980ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
57712025-07-17T11:35:49.988ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57722025-07-17T11:35:49.989ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 223.588768ms iteration = 22 status = {"physical_disks_added":0}
57732025-07-17T11:35:49.990ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
57742025-07-17T11:35:49.991ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
57752025-07-17T11:35:49.995ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
57762025-07-17T11:35:49.995ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
57772025-07-17T11:35:49.995ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
57782025-07-17T11:35:49.995ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
57792025-07-17T11:35:49.996ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
57802025-07-17T11:35:49.998ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 4a9e4c5e-728e-4850-bd58-14548c02bf6f dns_zone_name = oxide.test found_so_far = 4 version = 2
57812025-07-17T11:35:49.998ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
57822025-07-17T11:35:49.998ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-07-17 11:26:35.847594 UTC dns_group = external generation = 2
57832025-07-17T11:35:49.998ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 321.212055ms iteration = 11 status = {"generation":2}
57842025-07-17T11:35:49.999ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
57852025-07-17T11:35:49.999ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
57862025-07-17T11:35:50.002ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
57872025-07-17T11:35:50.002ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
57882025-07-17T11:35:50.005ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57892025-07-17T11:35:50.005ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57902025-07-17T11:35:50.005ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:372
57912025-07-17T11:35:50.005ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 239.755921ms iteration = 21 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
57922025-07-17T11:35:50.006ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 240.116585ms iteration = 21 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
57932025-07-17T11:35:50.008ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
57942025-07-17T11:35:50.009ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
57952025-07-17T11:35:50.009ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
57962025-07-17T11:35:50.009ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
57972025-07-17T11:35:50.009ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
57982025-07-17T11:35:50.011ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
57992025-07-17T11:35:50.011ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 245.663439ms iteration = 61 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
58002025-07-17T11:35:50.012ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
58012025-07-17T11:35:50.012ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
58022025-07-17T11:35:50.012ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58032025-07-17T11:35:50.012ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
58042025-07-17T11:35:50.016ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
58052025-07-17T11:35:50.016ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
58062025-07-17T11:35:50.016ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
58072025-07-17T11:35:50.016ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
58082025-07-17T11:35:50.016ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
58092025-07-17T11:35:50.016ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
58102025-07-17T11:35:50.017ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
58112025-07-17T11:35:50.019ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
58122025-07-17T11:35:50.020ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
58132025-07-17T11:35:50.022ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58142025-07-17T11:35:50.022ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58152025-07-17T11:35:50.025ZINFOnexus (BackgroundTasks): ensured all Crucible datasets present in inventory have database records background_task = blueprint_rendezvous file = nexus/reconfigurator/rendezvous/src/crucible_dataset.rs:113 num_already_exist = 9 num_inserted = 0 num_not_in_inventory = 0
58162025-07-17T11:35:50.025ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 340.073864ms iteration = 4 status = {"blueprint_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","inventory_collection_id":"bd15575d-5499-4736-988b-3d1df00d704b","stats":{"crucible_dataset":{"num_already_exist":9,"num_inserted":0,"num_not_in_inventory":0},"debug_dataset":{"num_already_exist":9,"num_already_tombstoned":0,"num_inserted":0,"num_not_in_inventory":0,"num_tombstoned":0}}}
58172025-07-17T11:35:50.027ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 0ae78f24-48db-4cf7-91cb-8d5311f2a17f, lookup_type: ById(0ae78f24-48db-4cf7-91cb-8d5311f2a17f) } result = Ok(())
58182025-07-17T11:35:50.027ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
58192025-07-17T11:35:50.027ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
58202025-07-17T11:35:50.029ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 263.93673ms iteration = 21 status = 0
58212025-07-17T11:35:50.030ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
58222025-07-17T11:35:50.030ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
58232025-07-17T11:35:50.031ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
58242025-07-17T11:35:50.037ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
58252025-07-17T11:35:50.037ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
58262025-07-17T11:35:50.040ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
58272025-07-17T11:35:50.040ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
58282025-07-17T11:35:50.042ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
58292025-07-17T11:35:50.043ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
58302025-07-17T11:35:50.057ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
58312025-07-17T11:35:50.058ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
58322025-07-17T11:35:50.060ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
58332025-07-17T11:35:50.060ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
58342025-07-17T11:35:50.060ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 294.590074ms iteration = 21 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
58352025-07-17T11:35:50.066ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
58362025-07-17T11:35:50.066ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
58372025-07-17T11:35:50.069ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
58382025-07-17T11:35:50.070ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
58392025-07-17T11:35:50.071ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 304.996921ms iteration = 21 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
58402025-07-17T11:35:50.072ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
58412025-07-17T11:35:50.073ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
58422025-07-17T11:35:50.075ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
58432025-07-17T11:35:50.075ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
58442025-07-17T11:35:50.077ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58452025-07-17T11:35:50.083ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
58462025-07-17T11:35:50.083ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
58472025-07-17T11:35:50.083ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
58482025-07-17T11:35:50.085ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
58492025-07-17T11:35:50.085ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
58502025-07-17T11:35:50.087ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
58512025-07-17T11:35:50.088ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
58522025-07-17T11:35:50.090ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58532025-07-17T11:35:50.095ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
58542025-07-17T11:35:50.095ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
58552025-07-17T11:35:50.095ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
58562025-07-17T11:35:50.103ZDEBGnexus (service-firewall-rule-progator): resolved names for firewall rules background_task = service_firewall_rule_propagation instance_interfaces = {} subnet_interfaces = {Name("external-dns"): [NetworkInterface { id: 82afb5bb-4e12-4b36-a03e-f32580a36236, kind: Service { id: 79e5c77d-18b3-4951-8d11-cfa3eae2b3e3 }, name: Name("external-dns-79e5c77d-18b3-4951-8d11-cfa3eae2b3e3"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 136, 29])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 949f6c29-3095-471f-b7dd-aa2aa681af87, kind: Service { id: 3dd3bf3b-7760-4750-b1d3-52ee777335f0 }, name: Name("external-dns-3dd3bf3b-7760-4750-b1d3-52ee777335f0"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 144, 121])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], Name("nexus"): [NetworkInterface { id: 1c4a40e7-c575-4aa4-8a05-4f7305d5faa6, kind: Service { id: 5e9f3641-e36d-4c7a-ab2f-ad78baee50f9 }, name: Name("nexus-5e9f3641-e36d-4c7a-ab2f-ad78baee50f9"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 225, 75])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 26ccceb4-722f-401c-9efa-2678860d14f3, kind: Service { id: f39d3e70-66e4-468a-a44d-746d22bea3a0 }, name: Name("nexus-f39d3e70-66e4-468a-a44d-746d22bea3a0"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 232, 228])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 89aa3989-5913-43b2-86dd-df3c95477576, kind: Service { id: 1017ad63-50a5-4825-ad3a-e3d3e6cffc2e }, name: Name("nexus-1017ad63-50a5-4825-ad3a-e3d3e6cffc2e"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 202, 53])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }]} subnet_networks = {Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })], Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })]} vpc_interfaces = {}
58572025-07-17T11:35:50.103ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
58582025-07-17T11:35:50.103ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
58592025-07-17T11:35:50.105ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58602025-07-17T11:35:50.108ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Fleet result = Ok(())
58612025-07-17T11:35:50.108ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
58622025-07-17T11:35:50.108ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
58632025-07-17T11:35:50.110ZDEBGnexus (service-firewall-rule-progator): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified. background_task = service_firewall_rule_propagation
58642025-07-17T11:35:50.110ZDEBGnexus (service-firewall-rule-progator): resolved firewall rules for sled agents background_task = service_firewall_rule_propagation sled_agent_rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 82afb5bb-4e12-4b36-a03e-f32580a36236, kind: Service { id: 79e5c77d-18b3-4951-8d11-cfa3eae2b3e3 }, name: Name("external-dns-79e5c77d-18b3-4951-8d11-cfa3eae2b3e3"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 136, 29])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 949f6c29-3095-471f-b7dd-aa2aa681af87, kind: Service { id: 3dd3bf3b-7760-4750-b1d3-52ee777335f0 }, name: Name("external-dns-3dd3bf3b-7760-4750-b1d3-52ee777335f0"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 144, 121])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 1c4a40e7-c575-4aa4-8a05-4f7305d5faa6, kind: Service { id: 5e9f3641-e36d-4c7a-ab2f-ad78baee50f9 }, name: Name("nexus-5e9f3641-e36d-4c7a-ab2f-ad78baee50f9"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 225, 75])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 26ccceb4-722f-401c-9efa-2678860d14f3, kind: Service { id: f39d3e70-66e4-468a-a44d-746d22bea3a0 }, name: Name("nexus-f39d3e70-66e4-468a-a44d-746d22bea3a0"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 232, 228])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 89aa3989-5913-43b2-86dd-df3c95477576, kind: Service { id: 1017ad63-50a5-4825-ad3a-e3d3e6cffc2e }, name: Name("nexus-1017ad63-50a5-4825-ad3a-e3d3e6cffc2e"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 202, 53])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 82afb5bb-4e12-4b36-a03e-f32580a36236, kind: Service { id: 79e5c77d-18b3-4951-8d11-cfa3eae2b3e3 }, name: Name("external-dns-79e5c77d-18b3-4951-8d11-cfa3eae2b3e3"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 136, 29])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 949f6c29-3095-471f-b7dd-aa2aa681af87, kind: Service { id: 3dd3bf3b-7760-4750-b1d3-52ee777335f0 }, name: Name("external-dns-3dd3bf3b-7760-4750-b1d3-52ee777335f0"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 144, 121])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp(Some(VpcFirewallIcmpFilter { icmp_type: 3, code: Some(IcmpParamRange { first: 3, last: 4 }) })), Icmp(Some(VpcFirewallIcmpFilter { icmp_type: 5, code: None })), Icmp(Some(VpcFirewallIcmpFilter { icmp_type: 11, code: None }))]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 1c4a40e7-c575-4aa4-8a05-4f7305d5faa6, kind: Service { id: 5e9f3641-e36d-4c7a-ab2f-ad78baee50f9 }, name: Name("nexus-5e9f3641-e36d-4c7a-ab2f-ad78baee50f9"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 225, 75])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 26ccceb4-722f-401c-9efa-2678860d14f3, kind: Service { id: f39d3e70-66e4-468a-a44d-746d22bea3a0 }, name: Name("nexus-f39d3e70-66e4-468a-a44d-746d22bea3a0"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 232, 228])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 89aa3989-5913-43b2-86dd-df3c95477576, kind: Service { id: 1017ad63-50a5-4825-ad3a-e3d3e6cffc2e }, name: Name("nexus-1017ad63-50a5-4825-ad3a-e3d3e6cffc2e"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 202, 53])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }]
58652025-07-17T11:35:50.111ZDEBGnexus (service-firewall-rule-progator): resolved 3 rules for sleds background_task = service_firewall_rule_propagation
58662025-07-17T11:35:50.122ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
58672025-07-17T11:35:50.149ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/all-sp-ids
58682025-07-17T11:35:50.150ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/all-sp-ids", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a27a6eb-f1cf-4372-b1ad-4fc8d32f06a9", "content-length": "78", "date": "Thu, 17 Jul 2025 11:35:50 GMT"} })
58692025-07-17T11:35:50.150ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = switch
58702025-07-17T11:35:50.150ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = sled
58712025-07-17T11:35:50.150ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 1 sp_type = sled
58722025-07-17T11:35:50.151ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = sled
58732025-07-17T11:35:50.151ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = switch
58742025-07-17T11:35:50.151ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 1 sp_type = sled
58752025-07-17T11:35:50.153ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 0 sp_type = sled
58762025-07-17T11:35:50.153ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 0 sp_type = switch
58772025-07-17T11:35:50.157ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 0 sp_type = sled
58782025-07-17T11:35:50.157ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = sled
58792025-07-17T11:35:50.157ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = sled
58802025-07-17T11:35:50.157ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 0 sp_type = switch
58812025-07-17T11:35:50.157ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = switch
58822025-07-17T11:35:50.157ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = switch
58832025-07-17T11:35:50.158ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 1 sp_type = sled
58842025-07-17T11:35:50.161ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 1 sp_type = sled
58852025-07-17T11:35:50.161ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 1 sp_type = sled
58862025-07-17T11:35:50.161ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 1 sp_type = sled
58872025-07-17T11:35:50.197ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1194 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 zone_address = fd00:1122:3344:101::2
58882025-07-17T11:35:50.197ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
58892025-07-17T11:35:50.197ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "39821593-3866-4640-86f8-ff270daba12b", "content-length": "26", "date": "Thu, 17 Jul 2025 11:35:49 GMT"} })
58902025-07-17T11:35:50.198ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1197 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
58912025-07-17T11:35:50.198ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1194 zone_address = fd00:1122:3344:101::2
58922025-07-17T11:35:50.198ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
58932025-07-17T11:35:50.198ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "06adef44-f9f2-4549-9d9c-5da6c797cecb", "content-length": "26", "date": "Thu, 17 Jul 2025 11:35:50 GMT"} })
58942025-07-17T11:35:50.198ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1197 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
58952025-07-17T11:35:50.201ZDEBGnexus (service-firewall-rule-progator): resolved sleds for vpc oxide-services background_task = service_firewall_rule_propagation vpc_to_sled = [Sled { identity: SledIdentity { id: 0ae78f24-48db-4cf7-91cb-8d5311f2a17f, time_created: 2025-07-17T11:26:26.884734Z, time_modified: 2025-07-17T11:26:26.884734Z }, time_deleted: None, rcgen: Generation(Generation(19)), rack_id: 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5, is_scrimlet: true, serial_number: "sock", part_number: "i86pc", revision: SqlU32(0), usable_hardware_threads: SqlU32(48), usable_physical_ram: ByteCount(ByteCount(102952026112)), reservoir_size: ByteCount(ByteCount(56094621696)), ip: fd00:1122:3344:101::1, port: SqlU16(12345), last_used_address: fd00:1122:3344:101::ffff, policy: InService, state: Active, sled_agent_gen: Generation(Generation(1)), repo_depot_port: SqlU16(12348) }]
58962025-07-17T11:35:50.201ZDEBGnexus (service-firewall-rule-progator): sending firewall rules to sled agents background_task = service_firewall_rule_propagation
58972025-07-17T11:35:50.204ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
58982025-07-17T11:35:50.204ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d836d52-644a-49f9-8e25-2efc8edc7887", "content-length": "691", "date": "Thu, 17 Jul 2025 11:35:50 GMT"} })
58992025-07-17T11:35:50.204ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
59002025-07-17T11:35:50.204ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
59012025-07-17T11:35:50.204ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
59022025-07-17T11:35:50.205ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
59032025-07-17T11:35:50.211ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
59042025-07-17T11:35:50.214ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 448.26094ms iteration = 21 status = {}
59052025-07-17T11:35:50.227ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 0 sp_type = switch start_ena = None
59062025-07-17T11:35:50.227ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
59072025-07-17T11:35:50.227ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 0 sp_type = sled start_ena = None
59082025-07-17T11:35:50.227ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
59092025-07-17T11:35:50.227ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7c0fbb8-3985-4136-b177-6eed53aae948", "content-length": "81", "date": "Thu, 17 Jul 2025 11:35:49 GMT"} })
59102025-07-17T11:35:50.228ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "2cee02eb-ffd4-4a76-ba79-de5c765cb85a", "content-length": "81", "date": "Thu, 17 Jul 2025 11:35:49 GMT"} })
59112025-07-17T11:35:50.230ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 1 sp_type = sled start_ena = None
59122025-07-17T11:35:50.230ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/1/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
59132025-07-17T11:35:50.230ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/1/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6bf3d03-65ca-4c53-a9af-5e8b887b69ee", "content-length": "81", "date": "Thu, 17 Jul 2025 11:35:49 GMT"} })
59142025-07-17T11:35:50.232ZDEBGnexus (BackgroundTasks): activation complete background_task = sp_ereport_ingester elapsed = 466.963166ms iteration = 21 status = {"errors":[],"sps":[]}
59152025-07-17T11:35:50.232ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1194 zone_address = fd00:1122:3344:101::2
59162025-07-17T11:35:50.232ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
59172025-07-17T11:35:50.233ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "a386b60d-cba4-4341-bea0-951b6104d0dd", "content-length": "26", "date": "Thu, 17 Jul 2025 11:35:50 GMT"} })
59182025-07-17T11:35:50.233ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1197 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
59192025-07-17T11:35:50.243ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
59202025-07-17T11:35:50.286ZINFOnexus (BackgroundTasks): VPC route manager sled 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
59212025-07-17T11:35:50.286ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
59222025-07-17T11:35:50.286ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "f55f934a-46f4-4375-ab18-d1b212f65ce9", "content-length": "372", "date": "Thu, 17 Jul 2025 11:35:50 GMT"} })
59232025-07-17T11:35:50.286ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
59242025-07-17T11:35:50.287ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
59252025-07-17T11:35:50.297ZINFOnexus (BackgroundTasks): instance reincarnation completed successfully background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:113 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0
59262025-07-17T11:35:50.297ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 611.35336ms iteration = 11 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
59272025-07-17T11:35:50.305ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
59282025-07-17T11:35:50.305ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
59292025-07-17T11:35:50.305ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "9a5f1792-08f3-4fab-9f13-6b65c449988a", "date": "Thu, 17 Jul 2025 11:35:50 GMT"} })
59302025-07-17T11:35:50.305ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
59312025-07-17T11:35:50.306ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
59322025-07-17T11:35:50.309ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
59332025-07-17T11:35:50.309ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
59342025-07-17T11:35:50.313ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
59352025-07-17T11:35:50.313ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
59362025-07-17T11:35:50.317ZINFOnexus (BackgroundTasks): VPC route manager sled 0ae78f24-48db-4cf7-91cb-8d5311f2a17f push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
59372025-07-17T11:35:50.317ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
59382025-07-17T11:35:50.317ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "868aeeb5-fdb9-4cba-b1e6-ee75fb8a1d5e", "date": "Thu, 17 Jul 2025 11:35:50 GMT"} })
59392025-07-17T11:35:50.320ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 554.40596ms iteration = 21 status = {}
59402025-07-17T11:35:50.351ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _cockroach._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_cockroach._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_cockroach._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 32221, target: Name("38f7b479-8745-4a60-a729-71a1d9fbc312.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_cockroach._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 32221, target: Name("67695497-2434-426f-85af-d580c466f11f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_cockroach._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 32221, target: Name("9ea5352d-d3fb-4fdd-b49a-b31749ecc4eb.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_cockroach._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 32221, target: Name("e0d30bf9-c801-46dd-a5ca-6bc4b5bd879a.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_cockroach._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 32221, target: Name("eaf60119-84d5-4be5-a81e-331ed99ebe93.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("eaf60119-84d5-4be5-a81e-331ed99ebe93.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::4)) }], valid_until: Instant { tv_sec: 3944, tv_nsec: 103253933 } })
59412025-07-17T11:35:50.352ZDEBGnexus (BackgroundTasks): begin collection background_task = inventory_collection
59422025-07-17T11:35:50.352ZDEBGnexus (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[fd00:1122:3344:101::2]:12225
59432025-07-17T11:35:50.352ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/ignition
59442025-07-17T11:35:50.353ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "91f9c2a1-383d-4340-9b3f-c1ca84cbdd3b", "content-length": "476", "date": "Thu, 17 Jul 2025 11:35:50 GMT"} })
59452025-07-17T11:35:50.353ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0
59462025-07-17T11:35:50.353ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0699c552-f861-4697-b344-da3d984fdcfa", "content-length": "734", "date": "Thu, 17 Jul 2025 11:35:49 GMT"} })
59472025-07-17T11:35:50.353ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=0
59482025-07-17T11:35:50.354ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "56768fc6-12fe-4814-aa77-61f1d742e47b", "content-length": "111", "date": "Thu, 17 Jul 2025 11:35:49 GMT"} })
59492025-07-17T11:35:50.354ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=1
59502025-07-17T11:35:50.354ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "376f97b0-b966-4973-82b9-4f709bc08135", "content-length": "111", "date": "Thu, 17 Jul 2025 11:35:50 GMT"} })
59512025-07-17T11:35:50.354ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=0
59522025-07-17T11:35:50.355ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4270dba4-64eb-424f-8662-b4da7f10ce80", "content-length": "170", "date": "Thu, 17 Jul 2025 11:35:49 GMT"} })
59532025-07-17T11:35:50.355ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=1
59542025-07-17T11:35:50.355ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "83ce3419-4668-4ca3-95fc-4ebd103cb27a", "content-length": "170", "date": "Thu, 17 Jul 2025 11:35:50 GMT"} })
59552025-07-17T11:35:50.355ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=0
59562025-07-17T11:35:50.356ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3660c59c-56fb-4913-bdad-98423d2ba645", "content-length": "179", "date": "Thu, 17 Jul 2025 11:35:50 GMT"} })
59572025-07-17T11:35:50.356ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=1
59582025-07-17T11:35:50.356ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3afd4c7c-cf54-4f6a-ba6c-9221ed0ea3f6", "content-length": "179", "date": "Thu, 17 Jul 2025 11:35:49 GMT"} })
59592025-07-17T11:35:50.356ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cmpa
59602025-07-17T11:35:50.356ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "81f7e7fb-49f8-4f24-9482-14a71199ec06", "content-length": "702", "date": "Thu, 17 Jul 2025 11:35:49 GMT"} })
59612025-07-17T11:35:50.356ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa
59622025-07-17T11:35:50.357ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7c46d2c-a698-4d80-b62a-37a930440246", "content-length": "727", "date": "Thu, 17 Jul 2025 11:35:49 GMT"} })
59632025-07-17T11:35:50.357ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa
59642025-07-17T11:35:50.357ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b99de95e-7e79-4e04-a2c0-e9e29a969474", "content-length": "729", "date": "Thu, 17 Jul 2025 11:35:49 GMT"} })
59652025-07-17T11:35:50.357ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa
59662025-07-17T11:35:50.357ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "61b6e66f-9298-4f9b-a4a8-c9336dca829c", "content-length": "728", "date": "Thu, 17 Jul 2025 11:35:49 GMT"} })
59672025-07-17T11:35:50.357ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0
59682025-07-17T11:35:50.358ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "75b0fdf9-3e91-4dc6-972d-2248a2daea2e", "content-length": "722", "date": "Thu, 17 Jul 2025 11:35:49 GMT"} })
59692025-07-17T11:35:50.358ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=0
59702025-07-17T11:35:50.358ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb326e07-5e85-4ef6-9e82-c8b5398dd359", "content-length": "109", "date": "Thu, 17 Jul 2025 11:35:49 GMT"} })
59712025-07-17T11:35:50.358ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=1
59722025-07-17T11:35:50.359ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8db4d2e2-da6c-44c0-ad13-1b50222f5678", "content-length": "109", "date": "Thu, 17 Jul 2025 11:35:49 GMT"} })
59732025-07-17T11:35:50.359ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=0
59742025-07-17T11:35:50.359ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3219eef2-3426-45f6-93e8-2b3e2627c133", "content-length": "169", "date": "Thu, 17 Jul 2025 11:35:50 GMT"} })
59752025-07-17T11:35:50.359ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=1
59762025-07-17T11:35:50.360ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "33936cb0-b783-4e00-8551-f2687272779d", "content-length": "169", "date": "Thu, 17 Jul 2025 11:35:50 GMT"} })
59772025-07-17T11:35:50.360ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=0
59782025-07-17T11:35:50.360ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b30af15-63f6-4dbe-880b-0bb403c1817e", "content-length": "178", "date": "Thu, 17 Jul 2025 11:35:49 GMT"} })
59792025-07-17T11:35:50.360ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=1
59802025-07-17T11:35:50.361ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "57fb1884-7476-4250-8ad7-582a6f807630", "content-length": "178", "date": "Thu, 17 Jul 2025 11:35:49 GMT"} })
59812025-07-17T11:35:50.361ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cmpa
59822025-07-17T11:35:50.361ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b6843d6-2715-4286-9145-13b22b78b643", "content-length": "702", "date": "Thu, 17 Jul 2025 11:35:49 GMT"} })
59832025-07-17T11:35:50.361ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa
59842025-07-17T11:35:50.361ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "659c77d9-98ce-4a4a-8096-b58672eff6e2", "content-length": "727", "date": "Thu, 17 Jul 2025 11:35:50 GMT"} })
59852025-07-17T11:35:50.361ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa
59862025-07-17T11:35:50.361ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "79693cad-e2f5-4c10-9069-73e772c3a5fa", "content-length": "729", "date": "Thu, 17 Jul 2025 11:35:50 GMT"} })
59872025-07-17T11:35:50.361ZDEBGnexus (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa
59882025-07-17T11:35:50.362ZDEBGnexus (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb6799aa-c4c5-40e4-b17f-f627985a00c9", "content-length": "728", "date": "Thu, 17 Jul 2025 11:35:50 GMT"} })
59892025-07-17T11:35:50.362ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
59902025-07-17T11:35:50.362ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
59912025-07-17T11:35:50.364ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59922025-07-17T11:35:50.366ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
59932025-07-17T11:35:50.366ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
59942025-07-17T11:35:50.366ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
59952025-07-17T11:35:50.368ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59962025-07-17T11:35:50.370ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
59972025-07-17T11:35:50.370ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
59982025-07-17T11:35:50.370ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
59992025-07-17T11:35:50.388ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
60002025-07-17T11:35:50.402ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
60012025-07-17T11:35:50.402ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
60022025-07-17T11:35:50.404ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60032025-07-17T11:35:50.406ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
60042025-07-17T11:35:50.406ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
60052025-07-17T11:35:50.406ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
60062025-07-17T11:35:50.408ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
60072025-07-17T11:35:50.408ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
60082025-07-17T11:35:50.409ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60092025-07-17T11:35:50.411ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
60102025-07-17T11:35:50.411ZDEBGnexus (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
60112025-07-17T11:35:50.411ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
60122025-07-17T11:35:50.414ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = tuf_artifact_replication body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/artifacts-config
60132025-07-17T11:35:50.414ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = tuf_artifact_replication result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/artifacts-config", status: 204, headers: {"x-request-id": "da4316dd-37c6-42f7-ad01-cbb9bd77f597", "date": "Thu, 17 Jul 2025 11:35:50 GMT"} })
60142025-07-17T11:35:50.414ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = tuf_artifact_replication body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/artifacts
60152025-07-17T11:35:50.414ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = tuf_artifact_replication result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/artifacts", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0eaf10f-90a8-4aa1-96eb-923a211fcc4e", "content-length": "26", "date": "Thu, 17 Jul 2025 11:35:50 GMT"} })
60162025-07-17T11:35:50.414ZINFOnexus (BackgroundTasks): Successfully got artifact list background_task = tuf_artifact_replication file = nexus/src/app/background/tasks/tuf_artifact_replication.rs:672 sled = http://[fd00:1122:3344:101::1]:12345
60172025-07-17T11:35:50.417ZDEBGnexus (BackgroundTasks): activation complete background_task = tuf_artifact_replication elapsed = 725.260572ms iteration = 3 status = {"generation":1,"last_run_counters":{"copy_err":0,"copy_ok":0,"list_err":0,"list_ok":1,"put_config_err":0,"put_config_ok":1,"put_err":0,"put_ok":0},"lifetime_counters":{"copy_err":0,"copy_ok":0,"list_err":0,"list_ok":2,"put_config_err":0,"put_config_ok":2,"put_err":0,"put_ok":0},"local_repos":0,"request_debug_ringbuf":[{"generation":1,"operation":"put_config","target_sled":"0ae78f24-48db-4cf7-91cb-8d5311f2a17f","time":"2025-07-17T11:30:50.105482430Z"},{"operation":"list","target_sled":"0ae78f24-48db-4cf7-91cb-8d5311f2a17f","time":"2025-07-17T11:30:50.118168848Z"},{"generation":1,"operation":"put_config","target_sled":"0ae78f24-48db-4cf7-91cb-8d5311f2a17f","time":"2025-07-17T11:35:50.414554757Z"},{"operation":"list","target_sled":"0ae78f24-48db-4cf7-91cb-8d5311f2a17f","time":"2025-07-17T11:35:50.414751240Z"}]}
60182025-07-17T11:35:50.449ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
60192025-07-17T11:35:50.449ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
60202025-07-17T11:35:50.451ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 774.912247ms iteration = 21 status = null
60212025-07-17T11:35:50.466ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 732.778631ms iteration = 21 status = {}
60222025-07-17T11:35:50.507ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
60232025-07-17T11:35:50.508ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "fe4e7d40-4030-4a5d-a4a7-75bb2217049e", "date": "Thu, 17 Jul 2025 11:35:50 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
60242025-07-17T11:35:50.508ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
60252025-07-17T11:35:50.510ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "07e99cec-913b-4467-964f-6d97a54df805", "date": "Thu, 17 Jul 2025 11:35:50 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
60262025-07-17T11:35:50.510ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
60272025-07-17T11:35:50.513ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
60282025-07-17T11:35:50.513ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 835.959231ms iteration = 13 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
60292025-07-17T11:35:50.535ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
60302025-07-17T11:35:50.535ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
60312025-07-17T11:35:50.538ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
60322025-07-17T11:35:50.538ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
60332025-07-17T11:35:50.540ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60342025-07-17T11:35:50.543ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Sled { parent: Fleet, key: 0ae78f24-48db-4cf7-91cb-8d5311f2a17f, lookup_type: ById(0ae78f24-48db-4cf7-91cb-8d5311f2a17f) } result = Ok(())
60352025-07-17T11:35:50.543ZDEBGnexus (service-firewall-rule-progator): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = service_firewall_rule_propagation body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules
60362025-07-17T11:35:50.544ZDEBGnexus (service-firewall-rule-progator): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = service_firewall_rule_propagation result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules", status: 204, headers: {"x-request-id": "aa880ba0-66eb-40fe-ad0a-34047363f938", "date": "Thu, 17 Jul 2025 11:35:50 GMT"} })
60372025-07-17T11:35:50.546ZINFOnexus (service-firewall-rule-progator): updated firewall rules on 1 sleds background_task = service_firewall_rule_propagation file = nexus/networking/src/firewall_rules.rs:491
60382025-07-17T11:35:50.546ZDEBGnexus (service-firewall-rule-progator): successfully propagated service firewall rules background_task = service_firewall_rule_propagation elapsed = 860.318352ms
60392025-07-17T11:35:50.546ZDEBGnexus (BackgroundTasks): activation complete background_task = service_firewall_rule_propagation elapsed = 860.330563ms iteration = 3 status = {}
60402025-07-17T11:35:50.578ZDEBGnexus (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[fd00:1122:3344:101::1]:12345
60412025-07-17T11:35:50.578ZDEBGnexus (BackgroundTasks): client request SledAgent = http://[fd00:1122:3344:101::1]:12345 background_task = inventory_collection body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/inventory
60422025-07-17T11:35:50.612ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
60432025-07-17T11:35:50.612ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "4e62452d-4640-4f38-b23d-4a1e5a3e0020", "date": "Thu, 17 Jul 2025 11:35:50 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
60442025-07-17T11:35:50.612ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
60452025-07-17T11:35:50.614ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "5ec5afd7-671d-4799-9065-e499e748e958", "date": "Thu, 17 Jul 2025 11:35:50 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
60462025-07-17T11:35:50.614ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
60472025-07-17T11:35:50.616ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "8c03b2d8-52fc-4bda-b903-a8fcb272c533", "date": "Thu, 17 Jul 2025 11:35:50 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
60482025-07-17T11:35:50.616ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
60492025-07-17T11:35:50.618ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
60502025-07-17T11:35:50.618ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 941.853407ms iteration = 13 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
60512025-07-17T11:35:50.662ZDEBGnexus (BackgroundTasks): client response SledAgent = http://[fd00:1122:3344:101::1]:12345 background_task = inventory_collection result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "65138d9e-215a-4999-bd97-4e83965fcc13", "content-length": "88892", "date": "Thu, 17 Jul 2025 11:35:50 GMT"} })
60522025-07-17T11:35:50.663ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
60532025-07-17T11:35:50.663ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
60542025-07-17T11:35:50.665ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 151caa7c-6a05-4995-94b4-7946d428117d, rack_id: 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5, switch_location: "switch0", port_name: Name(Name("qsfp0")), port_settings_id: Some(2e30c7ae-26da-46e4-9127-aac194921f20) } rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0
60552025-07-17T11:35:50.665ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
60562025-07-17T11:35:50.665ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
60572025-07-17T11:35:50.681ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1832 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = Name(Name("qsfp0"))
60582025-07-17T11:35:50.681ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1859 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0 switch_port_id = Name(Name("qsfp0")) switch_port_settings_id = 2e30c7ae-26da-46e4-9127-aac194921f20
60592025-07-17T11:35:50.681ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
60602025-07-17T11:35:50.681ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
60612025-07-17T11:35:50.681ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
60622025-07-17T11:35:50.681ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
60632025-07-17T11:35:50.681ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
60642025-07-17T11:35:50.681ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2006 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
60652025-07-17T11:35:50.682ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
60662025-07-17T11:35:50.682ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2042 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
60672025-07-17T11:35:50.682ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
60682025-07-17T11:35:50.682ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
60692025-07-17T11:35:50.682ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
60702025-07-17T11:35:50.685ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0
60712025-07-17T11:35:50.685ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
60722025-07-17T11:35:50.710ZDEBGnexus (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
60732025-07-17T11:35:50.710ZDEBGnexus (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
60742025-07-17T11:35:50.710ZDEBGnexus (BackgroundTasks): begin collection from CockroachDB nodes background_task = inventory_collection
60752025-07-17T11:35:50.710ZDEBGnexus (CockroachClusterAdminClient): client request body = None method = GET uri = http://[fd00:1122:3344:101::3]:32222/node/id
60762025-07-17T11:35:50.710ZDEBGnexus (CockroachClusterAdminClient): client request body = None method = GET uri = http://[fd00:1122:3344:101::5]:32222/node/id
60772025-07-17T11:35:50.710ZDEBGnexus (CockroachClusterAdminClient): client request body = None method = GET uri = http://[fd00:1122:3344:101::4]:32222/node/id
60782025-07-17T11:35:50.710ZDEBGnexus (CockroachClusterAdminClient): client request body = None method = GET uri = http://[fd00:1122:3344:101::6]:32222/node/id
60792025-07-17T11:35:50.710ZDEBGnexus (CockroachClusterAdminClient): client request body = None method = GET uri = http://[fd00:1122:3344:101::7]:32222/node/id
60802025-07-17T11:35:50.711ZDEBGnexus (CockroachClusterAdminClient): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::3]:32222/node/id", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee19aeae-48ed-49a4-9960-c129051e9bf5", "content-length": "64", "date": "Thu, 17 Jul 2025 11:35:50 GMT"} })
60812025-07-17T11:35:50.711ZDEBGnexus (CockroachClusterAdminClient): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::4]:32222/node/id", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf8b3680-686c-4750-86fe-dd7a39f29ccb", "content-length": "64", "date": "Thu, 17 Jul 2025 11:35:50 GMT"} })
60822025-07-17T11:35:50.711ZDEBGnexus (CockroachClusterAdminClient): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::6]:32222/node/id", status: 200, headers: {"content-type": "application/json", "x-request-id": "a02ebb95-3dde-4182-95c7-163806b4efe5", "content-length": "64", "date": "Thu, 17 Jul 2025 11:35:50 GMT"} })
60832025-07-17T11:35:50.711ZDEBGnexus (CockroachClusterAdminClient): client request body = None method = GET uri = http://[fd00:1122:3344:101::4]:32222/proxy/status/vars
60842025-07-17T11:35:50.711ZDEBGnexus (CockroachClusterAdminClient): client request body = None method = GET uri = http://[fd00:1122:3344:101::6]:32222/proxy/status/vars
60852025-07-17T11:35:50.711ZDEBGnexus (CockroachClusterAdminClient): client request body = None method = GET uri = http://[fd00:1122:3344:101::3]:32222/proxy/status/vars
60862025-07-17T11:35:50.711ZDEBGnexus (CockroachClusterAdminClient): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::5]:32222/node/id", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1e093b4-a177-4de6-b188-9d28c365f15e", "content-length": "64", "date": "Thu, 17 Jul 2025 11:35:50 GMT"} })
60872025-07-17T11:35:50.711ZDEBGnexus (CockroachClusterAdminClient): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::7]:32222/node/id", status: 200, headers: {"content-type": "application/json", "x-request-id": "863a9b6d-0696-4cdf-8d68-eef1067a57b1", "content-length": "64", "date": "Thu, 17 Jul 2025 11:35:50 GMT"} })
60882025-07-17T11:35:50.711ZDEBGnexus (CockroachClusterAdminClient): client request body = None method = GET uri = http://[fd00:1122:3344:101::7]:32222/proxy/status/vars
60892025-07-17T11:35:50.711ZDEBGnexus (CockroachClusterAdminClient): client request body = None method = GET uri = http://[fd00:1122:3344:101::5]:32222/proxy/status/vars
60902025-07-17T11:35:50.759ZDEBGnexus (CockroachClusterAdminClient): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::7]:32222/proxy/status/vars", status: 200, headers: {"content-type": "application/json", "x-request-id": "40707e49-dd66-4a08-9288-f03816871bab", "content-length": "395215", "date": "Thu, 17 Jul 2025 11:35:50 GMT"} })
60912025-07-17T11:35:50.768ZDEBGnexus (CockroachClusterAdminClient): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::4]:32222/proxy/status/vars", status: 200, headers: {"content-type": "application/json", "x-request-id": "74822ccd-3d94-45de-9fca-2542ff1fab18", "content-length": "389070", "date": "Thu, 17 Jul 2025 11:35:50 GMT"} })
60922025-07-17T11:35:50.769ZDEBGnexus (CockroachClusterAdminClient): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::6]:32222/proxy/status/vars", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d079009-4cc2-4c82-9520-a9ceb6615f0b", "content-length": "393679", "date": "Thu, 17 Jul 2025 11:35:50 GMT"} })
60932025-07-17T11:35:50.770ZDEBGnexus (CockroachClusterAdminClient): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::5]:32222/proxy/status/vars", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e41f978-36f8-44c1-a0dc-95415fa9a9a9", "content-length": "390264", "date": "Thu, 17 Jul 2025 11:35:50 GMT"} })
60942025-07-17T11:35:50.789ZDEBGnexus (CockroachClusterAdminClient): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::3]:32222/proxy/status/vars", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad9b5eea-1da2-4fb8-98b2-b8836189c5a1", "content-length": "413847", "date": "Thu, 17 Jul 2025 11:35:50 GMT"} })
60952025-07-17T11:35:50.792ZDEBGnexus (CockroachClusterAdminClient): Successfully fetched metrics from CockroachDB node node_id = 4
60962025-07-17T11:35:50.792ZDEBGnexus (CockroachClusterAdminClient): Successfully fetched metrics from CockroachDB node node_id = 2
60972025-07-17T11:35:50.792ZDEBGnexus (CockroachClusterAdminClient): Successfully fetched metrics from CockroachDB node node_id = 3
60982025-07-17T11:35:50.792ZDEBGnexus (CockroachClusterAdminClient): Successfully fetched metrics from CockroachDB node node_id = 5
60992025-07-17T11:35:50.792ZDEBGnexus (CockroachClusterAdminClient): Successfully fetched metrics from CockroachDB node node_id = 1
61002025-07-17T11:35:50.793ZDEBGnexus (BackgroundTasks): finished collection background_task = inventory_collection
61012025-07-17T11:35:50.794ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
61022025-07-17T11:35:50.794ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
61032025-07-17T11:35:50.796ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61042025-07-17T11:35:50.798ZDEBGnexus (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
61052025-07-17T11:35:50.804ZDEBGnexus (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
61062025-07-17T11:35:50.804ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
61072025-07-17T11:35:51.034ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "6a10ce20-2239-4099-af19-ee4421983e3d", "date": "Thu, 17 Jul 2025 11:35:50 GMT"} })
61082025-07-17T11:35:51.034ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch = Switch0
61092025-07-17T11:35:51.034ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
61102025-07-17T11:35:51.035ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "ede94257-55af-4edf-8536-06653fa4ae14", "content-length": "682", "date": "Thu, 17 Jul 2025 11:35:50 GMT"} })
61112025-07-17T11:35:51.035ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
61122025-07-17T11:35:51.035ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
61132025-07-17T11:35:51.039ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
61142025-07-17T11:35:51.040ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
61152025-07-17T11:35:51.045ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
61162025-07-17T11:35:51.045ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
61172025-07-17T11:35:51.048ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
61182025-07-17T11:35:51.048ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
61192025-07-17T11:35:51.057ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 1.291402355s iteration = 21 status = {}
61202025-07-17T11:35:51.069ZINFOnexus (BackgroundTasks): inserted inventory collection background_task = inventory_collection collection_id = 2a2fdc88-56f0-4da7-be8b-e61930cdd890 file = nexus/db-queries/src/db/datastore/inventory.rs:1442
61212025-07-17T11:35:51.069ZDEBGnexus (BackgroundTasks): inventory collection complete background_task = inventory_collection collection_id = 2a2fdc88-56f0-4da7-be8b-e61930cdd890 time_started = 2025-07-17 11:35:50.352665 UTC
61222025-07-17T11:35:51.069ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 23 reason = Dependency
61232025-07-17T11:35:51.069ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_planner iteration = 14 reason = Dependency
61242025-07-17T11:35:51.069ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
61252025-07-17T11:35:51.069ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 5 reason = Dependency
61262025-07-17T11:35:51.069ZDEBGnexus (BackgroundTasks): blueprint planning disabled, doing nothing background_task = blueprint_planner
61272025-07-17T11:35:51.069ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_planner elapsed = 11.14\u{b5}s iteration = 14 status = "Disabled"
61282025-07-17T11:35:51.069ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
61292025-07-17T11:35:51.069ZDEBGnexus (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.38588794s iteration = 2 status = {"collection_id":"2a2fdc88-56f0-4da7-be8b-e61930cdd890","time_done":"2025-07-17 11:35:50.793118 UTC","time_started":"2025-07-17 11:35:50.352665 UTC"}
61302025-07-17T11:35:51.070ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
61312025-07-17T11:35:51.070ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
61322025-07-17T11:35:51.072ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61332025-07-17T11:35:51.072ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61342025-07-17T11:35:51.075ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
61352025-07-17T11:35:51.075ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
61362025-07-17T11:35:51.075ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
61372025-07-17T11:35:51.076ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Inventory result = Ok(())
61382025-07-17T11:35:51.076ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
61392025-07-17T11:35:51.076ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
61402025-07-17T11:35:51.078ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
61412025-07-17T11:35:51.079ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
61422025-07-17T11:35:51.079ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
61432025-07-17T11:35:51.079ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
61442025-07-17T11:35:51.080ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61452025-07-17T11:35:51.083ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
61462025-07-17T11:35:51.083ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
61472025-07-17T11:35:51.083ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
61482025-07-17T11:35:51.123ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
61492025-07-17T11:35:51.123ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
61502025-07-17T11:35:51.125ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61512025-07-17T11:35:51.127ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
61522025-07-17T11:35:51.127ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
61532025-07-17T11:35:51.127ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
61542025-07-17T11:35:51.128ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61552025-07-17T11:35:51.130ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
61562025-07-17T11:35:51.130ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
61572025-07-17T11:35:51.130ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
61582025-07-17T11:35:51.133ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
61592025-07-17T11:35:51.133ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
61602025-07-17T11:35:51.134ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61612025-07-17T11:35:51.136ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
61622025-07-17T11:35:51.136ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
61632025-07-17T11:35:51.136ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
61642025-07-17T11:35:51.137ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61652025-07-17T11:35:51.139ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Fleet result = Ok(())
61662025-07-17T11:35:51.139ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_rendezvous roles = RoleSet { roles: {} }
61672025-07-17T11:35:51.139ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_rendezvous resource = Database result = Ok(())
61682025-07-17T11:35:51.142ZINFOnexus (BackgroundTasks): ensured all Crucible datasets present in inventory have database records background_task = blueprint_rendezvous file = nexus/reconfigurator/rendezvous/src/crucible_dataset.rs:113 num_already_exist = 9 num_inserted = 0 num_not_in_inventory = 0
61692025-07-17T11:35:51.142ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 72.733688ms iteration = 5 status = {"blueprint_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","inventory_collection_id":"f6be95fb-1b3a-46d7-b17c-66cde9225fb2","stats":{"crucible_dataset":{"num_already_exist":9,"num_inserted":0,"num_not_in_inventory":0},"debug_dataset":{"num_already_exist":9,"num_already_tombstoned":0,"num_inserted":0,"num_not_in_inventory":0,"num_tombstoned":0}}}
61702025-07-17T11:35:51.192ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 122.753799ms iteration = 23 status = {"physical_disks_added":0}
61712025-07-17T11:35:51.693ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 415 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = 7bd1ef89-cb6a-41ed-b3c3-a0e506a6093b response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
61722025-07-17T11:35:54.764ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 122 reason = Timeout
61732025-07-17T11:35:54.764ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
61742025-07-17T11:35:54.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
61752025-07-17T11:35:54.767ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61762025-07-17T11:35:54.770ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
61772025-07-17T11:35:54.770ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
61782025-07-17T11:35:54.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
61792025-07-17T11:35:54.772ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
61802025-07-17T11:35:54.772ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 7.394678ms iteration = 122 status = {"chicken_switches_updated":false}
61812025-07-17T11:35:59.765ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 62 reason = Timeout
61822025-07-17T11:35:59.765ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
61832025-07-17T11:35:59.765ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 62 reason = Timeout
61842025-07-17T11:35:59.765ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 123 reason = Timeout
61852025-07-17T11:35:59.765ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
61862025-07-17T11:35:59.765ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
61872025-07-17T11:35:59.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
61882025-07-17T11:35:59.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
61892025-07-17T11:35:59.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
61902025-07-17T11:35:59.768ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61912025-07-17T11:35:59.768ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61922025-07-17T11:35:59.772ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
61932025-07-17T11:35:59.772ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
61942025-07-17T11:35:59.772ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
61952025-07-17T11:35:59.772ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
61962025-07-17T11:35:59.772ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
61972025-07-17T11:35:59.772ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
61982025-07-17T11:35:59.774ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
61992025-07-17T11:35:59.774ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 8.763076ms iteration = 123 status = {"chicken_switches_updated":false}
62002025-07-17T11:35:59.775ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
62012025-07-17T11:35:59.775ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
62022025-07-17T11:35:59.776ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62032025-07-17T11:35:59.779ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
62042025-07-17T11:35:59.779ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
62052025-07-17T11:35:59.779ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
62062025-07-17T11:35:59.790ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
62072025-07-17T11:35:59.790ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 25.547547ms iteration = 62 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
62082025-07-17T11:35:59.800ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
62092025-07-17T11:35:59.800ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
62102025-07-17T11:35:59.834ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 69.280764ms iteration = 62 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
62112025-07-17T11:36:01.694ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 313 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = d2ce6827-51f5-4347-9c0c-f748cc24bbed response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
62122025-07-17T11:36:04.765ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 124 reason = Timeout
62132025-07-17T11:36:04.765ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
62142025-07-17T11:36:04.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
62152025-07-17T11:36:04.767ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62162025-07-17T11:36:04.770ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
62172025-07-17T11:36:04.770ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
62182025-07-17T11:36:04.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
62192025-07-17T11:36:04.772ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
62202025-07-17T11:36:04.772ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 7.205275ms iteration = 124 status = {"chicken_switches_updated":false}
62212025-07-17T11:36:09.765ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 63 reason = Timeout
62222025-07-17T11:36:09.765ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 63 reason = Timeout
62232025-07-17T11:36:09.765ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
62242025-07-17T11:36:09.765ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
62252025-07-17T11:36:09.765ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 125 reason = Timeout
62262025-07-17T11:36:09.765ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
62272025-07-17T11:36:09.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
62282025-07-17T11:36:09.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
62292025-07-17T11:36:09.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
62302025-07-17T11:36:09.767ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62312025-07-17T11:36:09.768ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62322025-07-17T11:36:09.771ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
62332025-07-17T11:36:09.771ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
62342025-07-17T11:36:09.771ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
62352025-07-17T11:36:09.771ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
62362025-07-17T11:36:09.771ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
62372025-07-17T11:36:09.772ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
62382025-07-17T11:36:09.773ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
62392025-07-17T11:36:09.774ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
62402025-07-17T11:36:09.774ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
62412025-07-17T11:36:09.774ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 8.588623ms iteration = 125 status = {"chicken_switches_updated":false}
62422025-07-17T11:36:09.775ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62432025-07-17T11:36:09.778ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
62442025-07-17T11:36:09.778ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
62452025-07-17T11:36:09.778ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
62462025-07-17T11:36:09.790ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
62472025-07-17T11:36:09.790ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 25.140362ms iteration = 63 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
62482025-07-17T11:36:09.801ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
62492025-07-17T11:36:09.802ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
62502025-07-17T11:36:09.803ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 37.869379ms iteration = 63 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
62512025-07-17T11:36:11.694ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 585 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = d6555adf-cde0-4d95-856c-a0b816c38ba2 response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
62522025-07-17T11:36:14.765ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 126 reason = Timeout
62532025-07-17T11:36:14.765ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
62542025-07-17T11:36:14.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
62552025-07-17T11:36:14.767ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62562025-07-17T11:36:14.770ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
62572025-07-17T11:36:14.770ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
62582025-07-17T11:36:14.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
62592025-07-17T11:36:14.772ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
62602025-07-17T11:36:14.772ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 7.198145ms iteration = 126 status = {"chicken_switches_updated":false}
62612025-07-17T11:36:19.676ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 22 reason = Timeout
62622025-07-17T11:36:19.677ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
62632025-07-17T11:36:19.677ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
62642025-07-17T11:36:19.679ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3973, tv_nsec: 430906786 } })
62652025-07-17T11:36:19.679ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1186
62662025-07-17T11:36:19.710ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1194 zone_address = fd00:1122:3344:101::2
62672025-07-17T11:36:19.710ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
62682025-07-17T11:36:19.710ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "eef40d28-586e-4e72-9a54-b6e603dec6a4", "content-length": "26", "date": "Thu, 17 Jul 2025 11:36:19 GMT"} })
62692025-07-17T11:36:19.710ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1197 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
62702025-07-17T11:36:19.711ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
62712025-07-17T11:36:19.734ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 22 reason = Timeout
62722025-07-17T11:36:19.734ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 22 reason = Timeout
62732025-07-17T11:36:19.734ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
62742025-07-17T11:36:19.734ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
62752025-07-17T11:36:19.737ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.353714ms iteration = 22 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
62762025-07-17T11:36:19.737ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3973, tv_nsec: 488907661 } })
62772025-07-17T11:36:19.737ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1186
62782025-07-17T11:36:19.748ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
62792025-07-17T11:36:19.748ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
62802025-07-17T11:36:19.751ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 74.268159ms iteration = 22 status = null
62812025-07-17T11:36:19.765ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 22 reason = Timeout
62822025-07-17T11:36:19.765ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
62832025-07-17T11:36:19.765ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 22 reason = Timeout
62842025-07-17T11:36:19.765ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
62852025-07-17T11:36:19.765ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
62862025-07-17T11:36:19.765ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 22 reason = Timeout
62872025-07-17T11:36:19.765ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
62882025-07-17T11:36:19.765ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 64 reason = Timeout
62892025-07-17T11:36:19.765ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
62902025-07-17T11:36:19.765ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
62912025-07-17T11:36:19.765ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 64 reason = Timeout
62922025-07-17T11:36:19.765ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
62932025-07-17T11:36:19.765ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 22 reason = Timeout
62942025-07-17T11:36:19.765ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
62952025-07-17T11:36:19.765ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 127 reason = Timeout
62962025-07-17T11:36:19.765ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
62972025-07-17T11:36:19.765ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 22 reason = Timeout
62982025-07-17T11:36:19.765ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
62992025-07-17T11:36:19.765ZDEBGnexus (BackgroundTasks): activating background_task = sp_ereport_ingester iteration = 22 reason = Timeout
63002025-07-17T11:36:19.765ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 22 reason = Timeout
63012025-07-17T11:36:19.765ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
63022025-07-17T11:36:19.765ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 22 reason = Timeout
63032025-07-17T11:36:19.765ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
63042025-07-17T11:36:19.765ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 22 reason = Timeout
63052025-07-17T11:36:19.765ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
63062025-07-17T11:36:19.765ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 22 reason = Timeout
63072025-07-17T11:36:19.765ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
63082025-07-17T11:36:19.765ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 22 reason = Timeout
63092025-07-17T11:36:19.765ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 24 reason = Timeout
63102025-07-17T11:36:19.765ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
63112025-07-17T11:36:19.765ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
63122025-07-17T11:36:19.765ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 22 reason = Timeout
63132025-07-17T11:36:19.765ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
63142025-07-17T11:36:19.765ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 22 reason = Timeout
63152025-07-17T11:36:19.765ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
63162025-07-17T11:36:19.765ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 22 reason = Timeout
63172025-07-17T11:36:19.765ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
63182025-07-17T11:36:19.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
63192025-07-17T11:36:19.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
63202025-07-17T11:36:19.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
63212025-07-17T11:36:19.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
63222025-07-17T11:36:19.765ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12225, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3973, tv_nsec: 517271935 } })
63232025-07-17T11:36:19.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
63242025-07-17T11:36:19.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
63252025-07-17T11:36:19.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
63262025-07-17T11:36:19.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
63272025-07-17T11:36:19.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
63282025-07-17T11:36:19.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
63292025-07-17T11:36:19.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
63302025-07-17T11:36:19.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
63312025-07-17T11:36:19.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
63322025-07-17T11:36:19.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
63332025-07-17T11:36:19.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
63342025-07-17T11:36:19.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
63352025-07-17T11:36:19.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
63362025-07-17T11:36:19.767ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63372025-07-17T11:36:19.768ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63382025-07-17T11:36:19.768ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
63392025-07-17T11:36:19.768ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
63402025-07-17T11:36:19.769ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63412025-07-17T11:36:19.770ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63422025-07-17T11:36:19.770ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63432025-07-17T11:36:19.771ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
63442025-07-17T11:36:19.771ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
63452025-07-17T11:36:19.771ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
63462025-07-17T11:36:19.771ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63472025-07-17T11:36:19.772ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
63482025-07-17T11:36:19.772ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
63492025-07-17T11:36:19.773ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
63502025-07-17T11:36:19.773ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
63512025-07-17T11:36:19.773ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
63522025-07-17T11:36:19.773ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
63532025-07-17T11:36:19.773ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
63542025-07-17T11:36:19.773ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
63552025-07-17T11:36:19.773ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
63562025-07-17T11:36:19.773ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63572025-07-17T11:36:19.773ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
63582025-07-17T11:36:19.773ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
63592025-07-17T11:36:19.774ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63602025-07-17T11:36:19.775ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
63612025-07-17T11:36:19.775ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
63622025-07-17T11:36:19.775ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
63632025-07-17T11:36:19.775ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
63642025-07-17T11:36:19.775ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
63652025-07-17T11:36:19.776ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
63662025-07-17T11:36:19.776ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
63672025-07-17T11:36:19.776ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 11.230948ms iteration = 127 status = {"chicken_switches_updated":false}
63682025-07-17T11:36:19.776ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
63692025-07-17T11:36:19.776ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
63702025-07-17T11:36:19.777ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
63712025-07-17T11:36:19.777ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
63722025-07-17T11:36:19.777ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
63732025-07-17T11:36:19.777ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
63742025-07-17T11:36:19.777ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63752025-07-17T11:36:19.777ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
63762025-07-17T11:36:19.778ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
63772025-07-17T11:36:19.778ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
63782025-07-17T11:36:19.778ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
63792025-07-17T11:36:19.780ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
63802025-07-17T11:36:19.780ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
63812025-07-17T11:36:19.780ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
63822025-07-17T11:36:19.781ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
63832025-07-17T11:36:19.781ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
63842025-07-17T11:36:19.784ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
63852025-07-17T11:36:19.785ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
63862025-07-17T11:36:19.785ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1194 zone_address = fd00:1122:3344:101::2
63872025-07-17T11:36:19.785ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
63882025-07-17T11:36:19.785ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb05f3df-d260-4040-ac24-3823db5e6b61", "content-length": "26", "date": "Thu, 17 Jul 2025 11:36:19 GMT"} })
63892025-07-17T11:36:19.785ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1197 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
63902025-07-17T11:36:19.786ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3973, tv_nsec: 537500862 } })
63912025-07-17T11:36:19.786ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1186 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
63922025-07-17T11:36:19.787ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
63932025-07-17T11:36:19.837ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
63942025-07-17T11:36:19.837ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
63952025-07-17T11:36:19.837ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
63962025-07-17T11:36:19.838ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
63972025-07-17T11:36:19.844ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
63982025-07-17T11:36:19.844ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
63992025-07-17T11:36:19.844ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
64002025-07-17T11:36:19.845ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
64012025-07-17T11:36:19.857ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
64022025-07-17T11:36:19.858ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
64032025-07-17T11:36:19.861ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 96.37699ms iteration = 22 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
64042025-07-17T11:36:19.863ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
64052025-07-17T11:36:19.863ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
64062025-07-17T11:36:19.865ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64072025-07-17T11:36:19.866ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:550 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
64082025-07-17T11:36:19.866ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 101.447947ms iteration = 22 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
64092025-07-17T11:36:19.867ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 101.788592ms iteration = 64 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
64102025-07-17T11:36:19.867ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64112025-07-17T11:36:19.867ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64122025-07-17T11:36:19.867ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 102.207927ms iteration = 22 status = {"errors":[],"garbage_collect_requested":[]}
64132025-07-17T11:36:19.869ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 103.700407ms iteration = 22 status = {"errors":[],"finish_invoked_ok":[]}
64142025-07-17T11:36:19.870ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
64152025-07-17T11:36:19.870ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
64162025-07-17T11:36:19.870ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
64172025-07-17T11:36:19.871ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
64182025-07-17T11:36:19.871ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
64192025-07-17T11:36:19.872ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
64202025-07-17T11:36:19.872ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 107.499917ms iteration = 22 status = {"errors":[],"requests_created_ok":[]}
64212025-07-17T11:36:19.874ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
64222025-07-17T11:36:19.874ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
64232025-07-17T11:36:19.874ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
64242025-07-17T11:36:19.877ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
64252025-07-17T11:36:19.878ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
64262025-07-17T11:36:19.884ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
64272025-07-17T11:36:19.884ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
64282025-07-17T11:36:19.888ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64292025-07-17T11:36:19.893ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 0ae78f24-48db-4cf7-91cb-8d5311f2a17f, lookup_type: ById(0ae78f24-48db-4cf7-91cb-8d5311f2a17f) } result = Ok(())
64302025-07-17T11:36:19.893ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
64312025-07-17T11:36:19.893ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
64322025-07-17T11:36:19.899ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 134.399481ms iteration = 22 status = 0
64332025-07-17T11:36:19.902ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 168.725585ms iteration = 22 status = {}
64342025-07-17T11:36:19.916ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1194 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 zone_address = fd00:1122:3344:101::2
64352025-07-17T11:36:19.916ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
64362025-07-17T11:36:19.916ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/all-sp-ids
64372025-07-17T11:36:19.916ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "6419c77c-ac83-4c9b-b9af-6dcfc21c664b", "content-length": "26", "date": "Thu, 17 Jul 2025 11:36:19 GMT"} })
64382025-07-17T11:36:19.917ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1197 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
64392025-07-17T11:36:19.917ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/all-sp-ids", status: 200, headers: {"content-type": "application/json", "x-request-id": "50b1b32e-9aa7-4b0d-9766-e76a63328f3d", "content-length": "78", "date": "Thu, 17 Jul 2025 11:36:19 GMT"} })
64402025-07-17T11:36:19.917ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = switch
64412025-07-17T11:36:19.917ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = sled
64422025-07-17T11:36:19.917ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 1 sp_type = sled
64432025-07-17T11:36:19.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = switch
64442025-07-17T11:36:19.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = sled
64452025-07-17T11:36:19.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 1 sp_type = sled
64462025-07-17T11:36:19.918ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
64472025-07-17T11:36:19.921ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 0 sp_type = sled
64482025-07-17T11:36:19.922ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 0 sp_type = switch
64492025-07-17T11:36:19.922ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
64502025-07-17T11:36:19.922ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 156.830078ms iteration = 64 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
64512025-07-17T11:36:19.922ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 1 sp_type = sled
64522025-07-17T11:36:19.926ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 0 sp_type = sled
64532025-07-17T11:36:19.926ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = sled
64542025-07-17T11:36:19.926ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = sled
64552025-07-17T11:36:19.927ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 1 sp_type = sled
64562025-07-17T11:36:19.927ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 1 sp_type = sled
64572025-07-17T11:36:19.927ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 1 sp_type = sled
64582025-07-17T11:36:19.928ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 0 sp_type = switch
64592025-07-17T11:36:19.928ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = switch
64602025-07-17T11:36:19.929ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = switch
64612025-07-17T11:36:19.929ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 163.629267ms iteration = 22 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
64622025-07-17T11:36:19.930ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
64632025-07-17T11:36:19.930ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
64642025-07-17T11:36:19.930ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
64652025-07-17T11:36:19.930ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 1 sp_type = sled start_ena = None
64662025-07-17T11:36:19.930ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/1/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
64672025-07-17T11:36:19.931ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
64682025-07-17T11:36:19.931ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/1/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f28868d-8bdf-46e7-8f64-d918c14668d4", "content-length": "81", "date": "Thu, 17 Jul 2025 11:36:19 GMT"} })
64692025-07-17T11:36:19.954ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:372
64702025-07-17T11:36:19.954ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 188.854ms iteration = 22 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
64712025-07-17T11:36:19.960ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
64722025-07-17T11:36:19.961ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
64732025-07-17T11:36:19.964ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64742025-07-17T11:36:19.966ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 200.592324ms iteration = 22 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
64752025-07-17T11:36:19.966ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
64762025-07-17T11:36:19.967ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
64772025-07-17T11:36:19.967ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
64782025-07-17T11:36:19.969ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64792025-07-17T11:36:19.972ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
64802025-07-17T11:36:19.972ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
64812025-07-17T11:36:19.972ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
64822025-07-17T11:36:19.997ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 0 sp_type = switch start_ena = None
64832025-07-17T11:36:19.997ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
64842025-07-17T11:36:19.998ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "87c116f4-7459-4f56-8c65-6a7846236e80", "content-length": "81", "date": "Thu, 17 Jul 2025 11:36:19 GMT"} })
64852025-07-17T11:36:20.001ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 0 sp_type = sled start_ena = None
64862025-07-17T11:36:20.001ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
64872025-07-17T11:36:20.001ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6da404d-0ae4-436e-ba3b-3237d015668f", "content-length": "81", "date": "Thu, 17 Jul 2025 11:36:19 GMT"} })
64882025-07-17T11:36:20.003ZDEBGnexus (BackgroundTasks): activation complete background_task = sp_ereport_ingester elapsed = 237.796605ms iteration = 22 status = {"errors":[],"sps":[]}
64892025-07-17T11:36:20.015ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
64902025-07-17T11:36:20.015ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
64912025-07-17T11:36:20.015ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 250.073277ms iteration = 22 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
64922025-07-17T11:36:20.018ZINFOnexus (BackgroundTasks): VPC route manager sled 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
64932025-07-17T11:36:20.018ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
64942025-07-17T11:36:20.019ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "6efa457f-3b3e-4677-8624-42120aaa584c", "content-length": "372", "date": "Thu, 17 Jul 2025 11:36:20 GMT"} })
64952025-07-17T11:36:20.019ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
64962025-07-17T11:36:20.019ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
64972025-07-17T11:36:20.068ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 302.504058ms iteration = 24 status = {"physical_disks_added":0}
64982025-07-17T11:36:20.076ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
64992025-07-17T11:36:20.076ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
65002025-07-17T11:36:20.076ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "e93a7a1e-ca26-4ed7-aeba-a95bbee62d0b", "date": "Thu, 17 Jul 2025 11:36:20 GMT"} })
65012025-07-17T11:36:20.076ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
65022025-07-17T11:36:20.077ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
65032025-07-17T11:36:20.079ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
65042025-07-17T11:36:20.079ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
65052025-07-17T11:36:20.083ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
65062025-07-17T11:36:20.083ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
65072025-07-17T11:36:20.087ZINFOnexus (BackgroundTasks): VPC route manager sled 0ae78f24-48db-4cf7-91cb-8d5311f2a17f push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
65082025-07-17T11:36:20.087ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
65092025-07-17T11:36:20.087ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "4fe0e4e0-28c7-4eed-86c8-b9006a74374b", "date": "Thu, 17 Jul 2025 11:36:20 GMT"} })
65102025-07-17T11:36:20.089ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 324.558818ms iteration = 22 status = {}
65112025-07-17T11:36:20.115ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
65122025-07-17T11:36:20.115ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
65132025-07-17T11:36:20.117ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 151caa7c-6a05-4995-94b4-7946d428117d, rack_id: 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5, switch_location: "switch0", port_name: Name(Name("qsfp0")), port_settings_id: Some(2e30c7ae-26da-46e4-9127-aac194921f20) } rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0
65142025-07-17T11:36:20.117ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
65152025-07-17T11:36:20.118ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
65162025-07-17T11:36:20.119ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
65172025-07-17T11:36:20.119ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "ecbf9ad7-db29-44c6-a776-bde3cb8bc1d8", "content-length": "691", "date": "Thu, 17 Jul 2025 11:36:20 GMT"} })
65182025-07-17T11:36:20.119ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
65192025-07-17T11:36:20.119ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
65202025-07-17T11:36:20.119ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
65212025-07-17T11:36:20.121ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 355.987823ms iteration = 22 status = {}
65222025-07-17T11:36:20.130ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1832 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = Name(Name("qsfp0"))
65232025-07-17T11:36:20.130ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1859 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0 switch_port_id = Name(Name("qsfp0")) switch_port_settings_id = 2e30c7ae-26da-46e4-9127-aac194921f20
65242025-07-17T11:36:20.130ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
65252025-07-17T11:36:20.130ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
65262025-07-17T11:36:20.130ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
65272025-07-17T11:36:20.130ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
65282025-07-17T11:36:20.130ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
65292025-07-17T11:36:20.130ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2006 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
65302025-07-17T11:36:20.131ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
65312025-07-17T11:36:20.131ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2042 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
65322025-07-17T11:36:20.131ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
65332025-07-17T11:36:20.131ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
65342025-07-17T11:36:20.131ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
65352025-07-17T11:36:20.134ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0
65362025-07-17T11:36:20.134ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
65372025-07-17T11:36:20.322ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "45b39f03-4cb5-47ef-b667-2fc44a83e4b4", "date": "Thu, 17 Jul 2025 11:36:20 GMT"} })
65382025-07-17T11:36:20.322ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch = Switch0
65392025-07-17T11:36:20.322ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
65402025-07-17T11:36:20.328ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "07c0ac5d-ffdf-4a8a-ac49-8808cf6155e7", "content-length": "682", "date": "Thu, 17 Jul 2025 11:36:20 GMT"} })
65412025-07-17T11:36:20.328ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
65422025-07-17T11:36:20.329ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
65432025-07-17T11:36:20.332ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
65442025-07-17T11:36:20.333ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
65452025-07-17T11:36:20.335ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
65462025-07-17T11:36:20.336ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
65472025-07-17T11:36:20.338ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
65482025-07-17T11:36:20.338ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
65492025-07-17T11:36:20.345ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 580.224239ms iteration = 22 status = {}
65502025-07-17T11:36:21.693ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 414 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = 888334c0-a14d-418f-9fd3-a181f5ce7148 response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
65512025-07-17T11:36:24.765ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 128 reason = Timeout
65522025-07-17T11:36:24.765ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
65532025-07-17T11:36:24.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
65542025-07-17T11:36:24.767ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65552025-07-17T11:36:24.770ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
65562025-07-17T11:36:24.770ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
65572025-07-17T11:36:24.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
65582025-07-17T11:36:24.773ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
65592025-07-17T11:36:24.773ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 7.367227ms iteration = 128 status = {"chicken_switches_updated":false}
65602025-07-17T11:36:29.765ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 65 reason = Timeout
65612025-07-17T11:36:29.765ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
65622025-07-17T11:36:29.765ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 65 reason = Timeout
65632025-07-17T11:36:29.765ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 129 reason = Timeout
65642025-07-17T11:36:29.765ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
65652025-07-17T11:36:29.765ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
65662025-07-17T11:36:29.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
65672025-07-17T11:36:29.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
65682025-07-17T11:36:29.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
65692025-07-17T11:36:29.767ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65702025-07-17T11:36:29.768ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65712025-07-17T11:36:29.768ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
65722025-07-17T11:36:29.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
65732025-07-17T11:36:29.771ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
65742025-07-17T11:36:29.771ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
65752025-07-17T11:36:29.771ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
65762025-07-17T11:36:29.771ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
65772025-07-17T11:36:29.771ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
65782025-07-17T11:36:29.771ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
65792025-07-17T11:36:29.773ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
65802025-07-17T11:36:29.773ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 8.182798ms iteration = 129 status = {"chicken_switches_updated":false}
65812025-07-17T11:36:29.773ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
65822025-07-17T11:36:29.773ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
65832025-07-17T11:36:29.775ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65842025-07-17T11:36:29.778ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
65852025-07-17T11:36:29.778ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
65862025-07-17T11:36:29.778ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
65872025-07-17T11:36:29.788ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
65882025-07-17T11:36:29.788ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 23.291577ms iteration = 65 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
65892025-07-17T11:36:29.806ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 40.890108ms iteration = 65 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
65902025-07-17T11:36:31.693ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 350 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = 17e5f302-fc3e-4d95-b8bd-d5e9fcd4468d response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
65912025-07-17T11:36:34.765ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 130 reason = Timeout
65922025-07-17T11:36:34.765ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
65932025-07-17T11:36:34.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
65942025-07-17T11:36:34.767ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65952025-07-17T11:36:34.770ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
65962025-07-17T11:36:34.770ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
65972025-07-17T11:36:34.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
65982025-07-17T11:36:34.771ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
65992025-07-17T11:36:34.771ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 6.640748ms iteration = 130 status = {"chicken_switches_updated":false}
66002025-07-17T11:36:39.764ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 66 reason = Timeout
66012025-07-17T11:36:39.764ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
66022025-07-17T11:36:39.764ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 66 reason = Timeout
66032025-07-17T11:36:39.764ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 131 reason = Timeout
66042025-07-17T11:36:39.764ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
66052025-07-17T11:36:39.764ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
66062025-07-17T11:36:39.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
66072025-07-17T11:36:39.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
66082025-07-17T11:36:39.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
66092025-07-17T11:36:39.767ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66102025-07-17T11:36:39.767ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66112025-07-17T11:36:39.768ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
66122025-07-17T11:36:39.768ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
66132025-07-17T11:36:39.769ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 5.079687ms iteration = 66 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
66142025-07-17T11:36:39.770ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
66152025-07-17T11:36:39.770ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
66162025-07-17T11:36:39.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
66172025-07-17T11:36:39.771ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
66182025-07-17T11:36:39.771ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
66192025-07-17T11:36:39.771ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
66202025-07-17T11:36:39.772ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
66212025-07-17T11:36:39.772ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 7.535369ms iteration = 131 status = {"chicken_switches_updated":false}
66222025-07-17T11:36:39.773ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
66232025-07-17T11:36:39.773ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
66242025-07-17T11:36:39.774ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66252025-07-17T11:36:39.777ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
66262025-07-17T11:36:39.777ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
66272025-07-17T11:36:39.777ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
66282025-07-17T11:36:39.786ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
66292025-07-17T11:36:39.786ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.217863ms iteration = 66 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
66302025-07-17T11:36:41.693ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 545 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = 912723f7-ecfa-487b-8605-12ebf4117cfc response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
66312025-07-17T11:36:44.765ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 132 reason = Timeout
66322025-07-17T11:36:44.765ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
66332025-07-17T11:36:44.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
66342025-07-17T11:36:44.767ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66352025-07-17T11:36:44.770ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
66362025-07-17T11:36:44.770ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
66372025-07-17T11:36:44.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
66382025-07-17T11:36:44.772ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
66392025-07-17T11:36:44.772ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 6.80205ms iteration = 132 status = {"chicken_switches_updated":false}
66402025-07-17T11:36:49.677ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 12 reason = Timeout
66412025-07-17T11:36:49.677ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 12 reason = Timeout
66422025-07-17T11:36:49.677ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 12 reason = Timeout
66432025-07-17T11:36:49.677ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
66442025-07-17T11:36:49.677ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-07-17 11:26:49.677928467 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
66452025-07-17T11:36:49.678ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 12 reason = Timeout
66462025-07-17T11:36:49.678ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
66472025-07-17T11:36:49.678ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 12 reason = Timeout
66482025-07-17T11:36:49.678ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 14 reason = Timeout
66492025-07-17T11:36:49.678ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
66502025-07-17T11:36:49.678ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 14 reason = Timeout
66512025-07-17T11:36:49.678ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
66522025-07-17T11:36:49.678ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 12 reason = Timeout
66532025-07-17T11:36:49.678ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
66542025-07-17T11:36:49.678ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
66552025-07-17T11:36:49.678ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 23 reason = Timeout
66562025-07-17T11:36:49.678ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
66572025-07-17T11:36:49.678ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
66582025-07-17T11:36:49.678ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
66592025-07-17T11:36:49.678ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
66602025-07-17T11:36:49.678ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
66612025-07-17T11:36:49.678ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("3144a75a-317f-4b68-93e2-a210ca1189d1.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("87408888-9df4-4e0c-bc8e-2f920d039100.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("ee4378c5-e5f2-41cf-8969-e5e03e1b9a41.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("ee4378c5-e5f2-41cf-8969-e5e03e1b9a41.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:3::1)) }], valid_until: Instant { tv_sec: 4003, tv_nsec: 429713990 } })
66622025-07-17T11:36:49.678ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("3dd3bf3b-7760-4750-b1d3-52ee777335f0.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("79e5c77d-18b3-4951-8d11-cfa3eae2b3e3.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("79e5c77d-18b3-4951-8d11-cfa3eae2b3e3.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::8)) }], valid_until: Instant { tv_sec: 4003, tv_nsec: 429730660 } })
66632025-07-17T11:36:49.678ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
66642025-07-17T11:36:49.679ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
66652025-07-17T11:36:49.679ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.226217ms iteration = 12 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
66662025-07-17T11:36:49.679ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } dns_group = internal
66672025-07-17T11:36:49.679ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.310997ms iteration = 12 status = {"addresses":["[fd00:1122:3344:1::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:3::1]:5353"]}
66682025-07-17T11:36:49.680ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 13 reason = Timeout
66692025-07-17T11:36:49.681ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:85 reason = blueprint disabled target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a
66702025-07-17T11:36:49.681ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 16.42\u{b5}s iteration = 13 status = {"enabled":false,"target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a"}
66712025-07-17T11:36:49.681ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 4003, tv_nsec: 432503436 } })
66722025-07-17T11:36:49.681ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1186
66732025-07-17T11:36:49.682ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
66742025-07-17T11:36:49.682ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
66752025-07-17T11:36:49.682ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
66762025-07-17T11:36:49.682ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
66772025-07-17T11:36:49.686ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_planner iteration = 15 reason = Timeout
66782025-07-17T11:36:49.686ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 12 reason = Timeout
66792025-07-17T11:36:49.686ZDEBGnexus (BackgroundTasks): blueprint planning disabled, doing nothing background_task = blueprint_planner
66802025-07-17T11:36:49.686ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:53
66812025-07-17T11:36:49.686ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_planner elapsed = 17.07\u{b5}s iteration = 15 status = "Disabled"
66822025-07-17T11:36:49.686ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
66832025-07-17T11:36:49.686ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 12 reason = Timeout
66842025-07-17T11:36:49.686ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
66852025-07-17T11:36:49.686ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
66862025-07-17T11:36:49.686ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66872025-07-17T11:36:49.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
66882025-07-17T11:36:49.688ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 12 reason = Timeout
66892025-07-17T11:36:49.688ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
66902025-07-17T11:36:49.688ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
66912025-07-17T11:36:49.688ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66922025-07-17T11:36:49.688ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66932025-07-17T11:36:49.689ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-07-17 11:26:49.677928467 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
66942025-07-17T11:36:49.689ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 11.252318ms iteration = 12 status = {"expiration":"2025-07-17T11:26:49.677928467Z","pruned":[]}
66952025-07-17T11:36:49.690ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
66962025-07-17T11:36:49.690ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
66972025-07-17T11:36:49.690ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
66982025-07-17T11:36:49.690ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66992025-07-17T11:36:49.693ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 12 reason = Timeout
67002025-07-17T11:36:49.693ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
67012025-07-17T11:36:49.693ZDEBGnexus (BackgroundTasks): activating background_task = webhook_deliverator iteration = 12 reason = Timeout
67022025-07-17T11:36:49.693ZDEBGnexus (BackgroundTasks): activating background_task = alert_dispatcher iteration = 12 reason = Timeout
67032025-07-17T11:36:49.693ZDEBGnexus (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} }
67042025-07-17T11:36:49.693ZDEBGnexus (BackgroundTasks): roles background_task = alert_dispatcher roles = RoleSet { roles: {} }
67052025-07-17T11:36:49.693ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
67062025-07-17T11:36:49.693ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
67072025-07-17T11:36:49.694ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database result = Ok(())
67082025-07-17T11:36:49.694ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
67092025-07-17T11:36:49.694ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = alert_dispatcher resource = Database result = Ok(())
67102025-07-17T11:36:49.694ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
67112025-07-17T11:36:49.694ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
67122025-07-17T11:36:49.694ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
67132025-07-17T11:36:49.694ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
67142025-07-17T11:36:49.694ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
67152025-07-17T11:36:49.695ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
67162025-07-17T11:36:49.695ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
67172025-07-17T11:36:49.695ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
67182025-07-17T11:36:49.695ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
67192025-07-17T11:36:49.695ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
67202025-07-17T11:36:49.696ZINFOnexus (BackgroundTasks): all webhook delivery tasks completed background_task = webhook_deliverator file = nexus/src/app/background/tasks/webhook_deliverator.rs:204 num_receivers = 0
67212025-07-17T11:36:49.696ZDEBGnexus (BackgroundTasks): activation complete background_task = webhook_deliverator elapsed = 3.03442ms iteration = 12 status = {"by_rx":{},"error":null}
67222025-07-17T11:36:49.697ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 3.614678ms iteration = 12 status = {"errors":[],"found_port_ok":[]}
67232025-07-17T11:36:49.697ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67242025-07-17T11:36:49.698ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67252025-07-17T11:36:49.701ZDEBGnexus (BackgroundTasks): roles background_task = alert_dispatcher roles = RoleSet { roles: {} }
67262025-07-17T11:36:49.701ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = alert_dispatcher resource = Database result = Ok(())
67272025-07-17T11:36:49.703ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
67282025-07-17T11:36:49.703ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
67292025-07-17T11:36:49.703ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
67302025-07-17T11:36:49.704ZDEBGnexus (BackgroundTasks): activation complete background_task = alert_dispatcher elapsed = 10.586199ms iteration = 12 status = {"dispatched":[],"errors":[],"glob_version":"161.0.0","globs_reprocessed":{},"no_receivers":[]}
67312025-07-17T11:36:49.704ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
67322025-07-17T11:36:49.704ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
67332025-07-17T11:36:49.704ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
67342025-07-17T11:36:49.734ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 23 reason = Timeout
67352025-07-17T11:36:49.734ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 23 reason = Timeout
67362025-07-17T11:36:49.734ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
67372025-07-17T11:36:49.735ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
67382025-07-17T11:36:49.738ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 4003, tv_nsec: 490122076 } })
67392025-07-17T11:36:49.739ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1186
67402025-07-17T11:36:49.739ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
67412025-07-17T11:36:49.739ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:170 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
67422025-07-17T11:36:49.740ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 52.032296ms iteration = 12 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
67432025-07-17T11:36:49.751ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 16.446797ms iteration = 23 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
67442025-07-17T11:36:49.752ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
67452025-07-17T11:36:49.752ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
67462025-07-17T11:36:49.752ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
67472025-07-17T11:36:49.752ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
67482025-07-17T11:36:49.752ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
67492025-07-17T11:36:49.754ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67502025-07-17T11:36:49.757ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
67512025-07-17T11:36:49.757ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 4a9e4c5e-728e-4850-bd58-14548c02bf6f dns_zone_name = oxide.test found_so_far = 0 version = 2
67522025-07-17T11:36:49.757ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
67532025-07-17T11:36:49.757ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
67542025-07-17T11:36:49.757ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
67552025-07-17T11:36:49.758ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
67562025-07-17T11:36:49.758ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
67572025-07-17T11:36:49.759ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
67582025-07-17T11:36:49.759ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 915fbba0-947f-4bf0-9f9f-48a79df81f7b dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
67592025-07-17T11:36:49.759ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
67602025-07-17T11:36:49.760ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
67612025-07-17T11:36:49.761ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67622025-07-17T11:36:49.764ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
67632025-07-17T11:36:49.764ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
67642025-07-17T11:36:49.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
67652025-07-17T11:36:49.765ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 23 reason = Timeout
67662025-07-17T11:36:49.765ZDEBGnexus (BackgroundTasks): activating background_task = sp_ereport_ingester iteration = 23 reason = Timeout
67672025-07-17T11:36:49.765ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
67682025-07-17T11:36:49.765ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 23 reason = Timeout
67692025-07-17T11:36:49.765ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
67702025-07-17T11:36:49.765ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 23 reason = Timeout
67712025-07-17T11:36:49.765ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 23 reason = Timeout
67722025-07-17T11:36:49.765ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
67732025-07-17T11:36:49.765ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
67742025-07-17T11:36:49.765ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 23 reason = Timeout
67752025-07-17T11:36:49.765ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 23 reason = Timeout
67762025-07-17T11:36:49.765ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
67772025-07-17T11:36:49.765ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
67782025-07-17T11:36:49.765ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 23 reason = Timeout
67792025-07-17T11:36:49.765ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 23 reason = Timeout
67802025-07-17T11:36:49.765ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
67812025-07-17T11:36:49.765ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
67822025-07-17T11:36:49.765ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 23 reason = Timeout
67832025-07-17T11:36:49.765ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
67842025-07-17T11:36:49.765ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 23 reason = Timeout
67852025-07-17T11:36:49.765ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 67 reason = Timeout
67862025-07-17T11:36:49.765ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
67872025-07-17T11:36:49.765ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
67882025-07-17T11:36:49.765ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 25 reason = Timeout
67892025-07-17T11:36:49.765ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
67902025-07-17T11:36:49.765ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 23 reason = Timeout
67912025-07-17T11:36:49.765ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
67922025-07-17T11:36:49.765ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 23 reason = Timeout
67932025-07-17T11:36:49.765ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
67942025-07-17T11:36:49.765ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 23 reason = Timeout
67952025-07-17T11:36:49.765ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
67962025-07-17T11:36:49.765ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
67972025-07-17T11:36:49.765ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
67982025-07-17T11:36:49.765ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 67 reason = Timeout
67992025-07-17T11:36:49.765ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
68002025-07-17T11:36:49.765ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 133 reason = Timeout
68012025-07-17T11:36:49.765ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
68022025-07-17T11:36:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
68032025-07-17T11:36:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
68042025-07-17T11:36:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
68052025-07-17T11:36:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
68062025-07-17T11:36:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
68072025-07-17T11:36:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
68082025-07-17T11:36:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
68092025-07-17T11:36:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
68102025-07-17T11:36:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
68112025-07-17T11:36:49.765ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12225, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 4003, tv_nsec: 517115912 } })
68122025-07-17T11:36:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
68132025-07-17T11:36:49.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
68142025-07-17T11:36:49.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
68152025-07-17T11:36:49.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
68162025-07-17T11:36:49.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
68172025-07-17T11:36:49.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
68182025-07-17T11:36:49.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
68192025-07-17T11:36:49.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
68202025-07-17T11:36:49.766ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
68212025-07-17T11:36:49.766ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:138 found = 0 not_ready_to_be_deleted = 0
68222025-07-17T11:36:49.766ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 80.106256ms iteration = 12 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
68232025-07-17T11:36:49.767ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
68242025-07-17T11:36:49.767ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
68252025-07-17T11:36:49.767ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68262025-07-17T11:36:49.768ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 915fbba0-947f-4bf0-9f9f-48a79df81f7b dns_zone_name = control-plane.oxide.internal found_so_far = 59 version = 1
68272025-07-17T11:36:49.768ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
68282025-07-17T11:36:49.768ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-07-17 11:26:34.922237 UTC dns_group = internal generation = 1
68292025-07-17T11:36:49.768ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 90.584494ms iteration = 12 status = {"generation":1}
68302025-07-17T11:36:49.769ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68312025-07-17T11:36:49.771ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68322025-07-17T11:36:49.772ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
68332025-07-17T11:36:49.772ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
68342025-07-17T11:36:49.772ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
68352025-07-17T11:36:49.773ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
68362025-07-17T11:36:49.773ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
68372025-07-17T11:36:49.773ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
68382025-07-17T11:36:49.773ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68392025-07-17T11:36:49.774ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
68402025-07-17T11:36:49.774ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
68412025-07-17T11:36:49.775ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
68422025-07-17T11:36:49.775ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
68432025-07-17T11:36:49.775ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
68442025-07-17T11:36:49.776ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
68452025-07-17T11:36:49.776ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
68462025-07-17T11:36:49.777ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
68472025-07-17T11:36:49.778ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68482025-07-17T11:36:49.778ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
68492025-07-17T11:36:49.778ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
68502025-07-17T11:36:49.779ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68512025-07-17T11:36:49.782ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
68522025-07-17T11:36:49.782ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
68532025-07-17T11:36:49.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
68542025-07-17T11:36:49.783ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
68552025-07-17T11:36:49.783ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
68562025-07-17T11:36:49.783ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
68572025-07-17T11:36:49.792ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-07-17T11:25:46.964149Z, time_modified: 2025-07-17T11:25:46.964149Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 1fa836d9-7854-4ada-8c89-c26d922646c6, db_silo: Silo { identity: SiloIdentity { id: 1fa836d9-7854-4ada-8c89-c26d922646c6, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-07-17T11:26:35.643175Z, time_modified: 2025-07-17T11:26:35.643175Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "d7c3455b70a66c86ec2a7e597ca46fe0ea0c49dde5c1363f198c20aa77d58442" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 1fa836d9-7854-4ada-8c89-c26d922646c6, db_silo: Silo { identity: SiloIdentity { id: 1fa836d9-7854-4ada-8c89-c26d922646c6, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-07-17T11:26:35.643175Z, time_modified: 2025-07-17T11:26:35.643175Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "d7c3455b70a66c86ec2a7e597ca46fe0ea0c49dde5c1363f198c20aa77d58442" }] }) }
68582025-07-17T11:36:49.792ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 115.023606ms iteration = 12 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"1fa836d9-7854-4ada-8c89-c26d922646c6","tls_certs":["d7c3455b70a66c86ec2a7e597ca46fe0ea0c49dde5c1363f198c20aa77d58442"]}},"default_endpoint":{"silo_id":"1fa836d9-7854-4ada-8c89-c26d922646c6","tls_certs":["d7c3455b70a66c86ec2a7e597ca46fe0ea0c49dde5c1363f198c20aa77d58442"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
68592025-07-17T11:36:49.829ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68602025-07-17T11:36:49.834ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
68612025-07-17T11:36:49.835ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
68622025-07-17T11:36:49.835ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
68632025-07-17T11:36:49.838ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 4a9e4c5e-728e-4850-bd58-14548c02bf6f dns_zone_name = oxide.test found_so_far = 4 version = 2
68642025-07-17T11:36:49.838ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
68652025-07-17T11:36:49.838ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-07-17 11:26:35.847594 UTC dns_group = external generation = 2
68662025-07-17T11:36:49.838ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 160.954032ms iteration = 12 status = {"generation":2}
68672025-07-17T11:36:49.844ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
68682025-07-17T11:36:49.845ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
68692025-07-17T11:36:49.845ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68702025-07-17T11:36:49.846ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 81.361942ms iteration = 23 status = {"errors":[],"finish_invoked_ok":[]}
68712025-07-17T11:36:49.851ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
68722025-07-17T11:36:49.851ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
68732025-07-17T11:36:49.852ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
68742025-07-17T11:36:49.853ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68752025-07-17T11:36:49.855ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
68762025-07-17T11:36:49.856ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
68772025-07-17T11:36:49.857ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 91.926332ms iteration = 23 status = {"errors":[],"requests_created_ok":[]}
68782025-07-17T11:36:49.858ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
68792025-07-17T11:36:49.858ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
68802025-07-17T11:36:49.858ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68812025-07-17T11:36:49.859ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68822025-07-17T11:36:49.859ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
68832025-07-17T11:36:49.861ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:550 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
68842025-07-17T11:36:49.861ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 96.30737ms iteration = 23 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
68852025-07-17T11:36:49.862ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 97.464705ms iteration = 23 status = {"errors":[],"garbage_collect_requested":[]}
68862025-07-17T11:36:49.863ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
68872025-07-17T11:36:49.863ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
68882025-07-17T11:36:49.863ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
68892025-07-17T11:36:49.864ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
68902025-07-17T11:36:49.864ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
68912025-07-17T11:36:49.865ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
68922025-07-17T11:36:49.870ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
68932025-07-17T11:36:49.870ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
68942025-07-17T11:36:49.872ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
68952025-07-17T11:36:49.872ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 106.580656ms iteration = 133 status = {"chicken_switches_updated":false}
68962025-07-17T11:36:49.879ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
68972025-07-17T11:36:49.879ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
68982025-07-17T11:36:49.880ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
68992025-07-17T11:36:49.880ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
69002025-07-17T11:36:49.881ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 4003, tv_nsec: 633126341 } })
69012025-07-17T11:36:49.882ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
69022025-07-17T11:36:49.882ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
69032025-07-17T11:36:49.882ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
69042025-07-17T11:36:49.882ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1186 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
69052025-07-17T11:36:49.882ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
69062025-07-17T11:36:49.887ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
69072025-07-17T11:36:49.887ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
69082025-07-17T11:36:49.902ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1194 zone_address = fd00:1122:3344:101::2
69092025-07-17T11:36:49.902ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
69102025-07-17T11:36:49.902ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d49fe9a-aeb5-4194-b254-9025ea96e683", "content-length": "26", "date": "Thu, 17 Jul 2025 11:36:49 GMT"} })
69112025-07-17T11:36:49.902ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1197 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
69122025-07-17T11:36:49.904ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
69132025-07-17T11:36:49.906ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
69142025-07-17T11:36:49.907ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
69152025-07-17T11:36:49.909ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 144.642397ms iteration = 23 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
69162025-07-17T11:36:49.910ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
69172025-07-17T11:36:49.910ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
69182025-07-17T11:36:49.913ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69192025-07-17T11:36:49.914ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
69202025-07-17T11:36:49.914ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
69212025-07-17T11:36:49.925ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 0ae78f24-48db-4cf7-91cb-8d5311f2a17f, lookup_type: ById(0ae78f24-48db-4cf7-91cb-8d5311f2a17f) } result = Ok(())
69222025-07-17T11:36:49.925ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
69232025-07-17T11:36:49.925ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
69242025-07-17T11:36:49.929ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 163.884181ms iteration = 23 status = 0
69252025-07-17T11:36:49.930ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
69262025-07-17T11:36:49.930ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
69272025-07-17T11:36:49.933ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69282025-07-17T11:36:49.933ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69292025-07-17T11:36:49.936ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 170.787632ms iteration = 23 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
69302025-07-17T11:36:49.937ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
69312025-07-17T11:36:49.937ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
69322025-07-17T11:36:49.937ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
69332025-07-17T11:36:49.939ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
69342025-07-17T11:36:49.939ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
69352025-07-17T11:36:49.940ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
69362025-07-17T11:36:49.949ZINFOnexus (BackgroundTasks): VPC route manager sled 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
69372025-07-17T11:36:49.949ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
69382025-07-17T11:36:49.949ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "b354b105-9f9e-400f-ad57-59f51afe0ee0", "content-length": "372", "date": "Thu, 17 Jul 2025 11:36:49 GMT"} })
69392025-07-17T11:36:49.949ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
69402025-07-17T11:36:49.950ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
69412025-07-17T11:36:49.951ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69422025-07-17T11:36:49.957ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
69432025-07-17T11:36:49.957ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
69442025-07-17T11:36:49.957ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 192.095793ms iteration = 67 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
69452025-07-17T11:36:49.957ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
69462025-07-17T11:36:49.963ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
69472025-07-17T11:36:49.963ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 198.285714ms iteration = 67 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
69482025-07-17T11:36:49.980ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
69492025-07-17T11:36:49.981ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1194 zone_address = fd00:1122:3344:101::2
69502025-07-17T11:36:49.981ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
69512025-07-17T11:36:49.981ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
69522025-07-17T11:36:49.981ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
69532025-07-17T11:36:49.981ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
69542025-07-17T11:36:49.981ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "24496405-9420-471b-ae27-264cdbe1f314", "content-length": "26", "date": "Thu, 17 Jul 2025 11:36:49 GMT"} })
69552025-07-17T11:36:49.981ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1197 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
69562025-07-17T11:36:49.982ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
69572025-07-17T11:36:49.986ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
69582025-07-17T11:36:50.008ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 243.376199ms iteration = 23 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
69592025-07-17T11:36:50.020ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
69602025-07-17T11:36:50.021ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
69612025-07-17T11:36:50.027ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:372
69622025-07-17T11:36:50.027ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 261.861461ms iteration = 23 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
69632025-07-17T11:36:50.053ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
69642025-07-17T11:36:50.053ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
69652025-07-17T11:36:50.053ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 288.182368ms iteration = 23 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
69662025-07-17T11:36:50.057ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
69672025-07-17T11:36:50.057ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
69682025-07-17T11:36:50.057ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "8a8b3705-996e-4de5-bc0b-76eaa2fd1643", "date": "Thu, 17 Jul 2025 11:36:50 GMT"} })
69692025-07-17T11:36:50.057ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
69702025-07-17T11:36:50.058ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
69712025-07-17T11:36:50.061ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
69722025-07-17T11:36:50.061ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
69732025-07-17T11:36:50.064ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
69742025-07-17T11:36:50.064ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
69752025-07-17T11:36:50.069ZINFOnexus (BackgroundTasks): VPC route manager sled 0ae78f24-48db-4cf7-91cb-8d5311f2a17f push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
69762025-07-17T11:36:50.069ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
69772025-07-17T11:36:50.069ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "07bb4749-cb02-4280-9d37-7b0938417965", "date": "Thu, 17 Jul 2025 11:36:50 GMT"} })
69782025-07-17T11:36:50.072ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 306.613691ms iteration = 23 status = {}
69792025-07-17T11:36:50.091ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/all-sp-ids
69802025-07-17T11:36:50.092ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/all-sp-ids", status: 200, headers: {"content-type": "application/json", "x-request-id": "4359e524-83aa-4662-b415-f6e6b32ef89c", "content-length": "78", "date": "Thu, 17 Jul 2025 11:36:49 GMT"} })
69812025-07-17T11:36:50.092ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = switch
69822025-07-17T11:36:50.092ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = sled
69832025-07-17T11:36:50.092ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 1 sp_type = sled
69842025-07-17T11:36:50.092ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 1 sp_type = sled
69852025-07-17T11:36:50.092ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = switch
69862025-07-17T11:36:50.092ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = sled
69872025-07-17T11:36:50.094ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 1 sp_type = sled
69882025-07-17T11:36:50.095ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 0 sp_type = switch
69892025-07-17T11:36:50.096ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 0 sp_type = sled
69902025-07-17T11:36:50.097ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 1 sp_type = sled
69912025-07-17T11:36:50.097ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 1 sp_type = sled
69922025-07-17T11:36:50.097ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 1 sp_type = sled
69932025-07-17T11:36:50.098ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 0 sp_type = switch
69942025-07-17T11:36:50.098ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = switch
69952025-07-17T11:36:50.098ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = switch
69962025-07-17T11:36:50.099ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 1 sp_type = sled start_ena = None
69972025-07-17T11:36:50.099ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/1/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
69982025-07-17T11:36:50.100ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/1/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "657c7bd9-47f3-4d0b-91d7-feafba7012a8", "content-length": "81", "date": "Thu, 17 Jul 2025 11:36:49 GMT"} })
69992025-07-17T11:36:50.100ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 0 sp_type = sled
70002025-07-17T11:36:50.100ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = sled
70012025-07-17T11:36:50.100ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = sled
70022025-07-17T11:36:50.103ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 0 sp_type = sled start_ena = None
70032025-07-17T11:36:50.103ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
70042025-07-17T11:36:50.106ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1da4317-6adf-4c3a-8862-15e8410b225f", "content-length": "81", "date": "Thu, 17 Jul 2025 11:36:49 GMT"} })
70052025-07-17T11:36:50.106ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
70062025-07-17T11:36:50.106ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
70072025-07-17T11:36:50.108ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70082025-07-17T11:36:50.112ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
70092025-07-17T11:36:50.112ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
70102025-07-17T11:36:50.113ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
70112025-07-17T11:36:50.115ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70122025-07-17T11:36:50.118ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
70132025-07-17T11:36:50.118ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
70142025-07-17T11:36:50.118ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
70152025-07-17T11:36:50.121ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
70162025-07-17T11:36:50.138ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 373.410552ms iteration = 25 status = {"physical_disks_added":0}
70172025-07-17T11:36:50.145ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 0 sp_type = switch start_ena = None
70182025-07-17T11:36:50.145ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
70192025-07-17T11:36:50.146ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1194 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 zone_address = fd00:1122:3344:101::2
70202025-07-17T11:36:50.146ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
70212025-07-17T11:36:50.146ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ae8e5c6-b1cf-4411-bd25-110b5c287ec6", "content-length": "81", "date": "Thu, 17 Jul 2025 11:36:49 GMT"} })
70222025-07-17T11:36:50.146ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "12b4fbfd-6a24-4491-a424-772a94d7bb3f", "content-length": "26", "date": "Thu, 17 Jul 2025 11:36:49 GMT"} })
70232025-07-17T11:36:50.146ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1197 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
70242025-07-17T11:36:50.150ZDEBGnexus (BackgroundTasks): activation complete background_task = sp_ereport_ingester elapsed = 385.3688ms iteration = 23 status = {"errors":[],"sps":[]}
70252025-07-17T11:36:50.151ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
70262025-07-17T11:36:50.176ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
70272025-07-17T11:36:50.177ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
70282025-07-17T11:36:50.178ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 500.527018ms iteration = 23 status = null
70292025-07-17T11:36:50.183ZINFOnexus (BackgroundTasks): instance reincarnation completed successfully background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:113 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0
70302025-07-17T11:36:50.183ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 496.537076ms iteration = 12 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
70312025-07-17T11:36:50.196ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 461.933209ms iteration = 23 status = {}
70322025-07-17T11:36:50.250ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
70332025-07-17T11:36:50.250ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "9cbfcdb2-82ed-41de-b230-cf365f460db4", "date": "Thu, 17 Jul 2025 11:36:50 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
70342025-07-17T11:36:50.250ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
70352025-07-17T11:36:50.252ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "18e1ba1e-10c9-42b4-9e4c-1af8e1e5cd71", "date": "Thu, 17 Jul 2025 11:36:50 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
70362025-07-17T11:36:50.252ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
70372025-07-17T11:36:50.255ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
70382025-07-17T11:36:50.255ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 577.068527ms iteration = 14 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
70392025-07-17T11:36:50.284ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
70402025-07-17T11:36:50.301ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
70412025-07-17T11:36:50.301ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6e47177-e9f1-4ffe-a6f1-588113db7ed6", "content-length": "691", "date": "Thu, 17 Jul 2025 11:36:50 GMT"} })
70422025-07-17T11:36:50.301ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
70432025-07-17T11:36:50.301ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
70442025-07-17T11:36:50.301ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
70452025-07-17T11:36:50.303ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 538.55825ms iteration = 23 status = {}
70462025-07-17T11:36:50.353ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
70472025-07-17T11:36:50.354ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "a73ee928-104e-446a-91fb-62f7acf0f1c4", "date": "Thu, 17 Jul 2025 11:36:50 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
70482025-07-17T11:36:50.354ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
70492025-07-17T11:36:50.355ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "20b1dadc-6150-492b-a080-53a5a3d87a74", "date": "Thu, 17 Jul 2025 11:36:50 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
70502025-07-17T11:36:50.355ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
70512025-07-17T11:36:50.357ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "1d4cc075-33c9-4c49-8217-5d24d9cb71cd", "date": "Thu, 17 Jul 2025 11:36:50 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
70522025-07-17T11:36:50.357ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
70532025-07-17T11:36:50.360ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
70542025-07-17T11:36:50.360ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 682.344015ms iteration = 14 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
70552025-07-17T11:36:50.423ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
70562025-07-17T11:36:50.423ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
70572025-07-17T11:36:50.425ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 151caa7c-6a05-4995-94b4-7946d428117d, rack_id: 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5, switch_location: "switch0", port_name: Name(Name("qsfp0")), port_settings_id: Some(2e30c7ae-26da-46e4-9127-aac194921f20) } rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0
70582025-07-17T11:36:50.425ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
70592025-07-17T11:36:50.426ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
70602025-07-17T11:36:50.438ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1832 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = Name(Name("qsfp0"))
70612025-07-17T11:36:50.438ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1859 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0 switch_port_id = Name(Name("qsfp0")) switch_port_settings_id = 2e30c7ae-26da-46e4-9127-aac194921f20
70622025-07-17T11:36:50.438ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
70632025-07-17T11:36:50.438ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
70642025-07-17T11:36:50.438ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
70652025-07-17T11:36:50.438ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
70662025-07-17T11:36:50.438ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
70672025-07-17T11:36:50.438ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2006 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
70682025-07-17T11:36:50.439ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
70692025-07-17T11:36:50.439ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2042 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
70702025-07-17T11:36:50.439ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
70712025-07-17T11:36:50.439ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
70722025-07-17T11:36:50.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
70732025-07-17T11:36:50.441ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0
70742025-07-17T11:36:50.441ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
70752025-07-17T11:36:50.603ZDEBGnexus (producer-registration-task): registering as oximeter producer with Nexus registration_count = 4
70762025-07-17T11:36:50.603ZDEBGnexus (producer-registration-task): will register with Nexus at [fd00:1122:3344:101::b]:12221
70772025-07-17T11:36:50.634ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::b]:12221/metrics/producers
70782025-07-17T11:36:50.634ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::b]:34176
70792025-07-17T11:36:50.634ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:34176 req_id = 37cee41b-7170-4a71-88b0-f56b8aa673aa roles = RoleSet { roles: {} } uri = /metrics/producers
70802025-07-17T11:36:50.634ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:34176 req_id = 37cee41b-7170-4a71-88b0-f56b8aa673aa resource = Database result = Ok(()) uri = /metrics/producers
70812025-07-17T11:36:50.682ZINFOnexus: assigned collector to new producer collector_id = 3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d file = nexus/src/app/oximeter.rs:76 producer_id = 5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
70822025-07-17T11:36:50.682ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 48152 local_addr = [fd00:1122:3344:101::b]:12221 method = POST remote_addr = [fd00:1122:3344:101::b]:34176 req_id = 37cee41b-7170-4a71-88b0-f56b8aa673aa response_code = 201 uri = /metrics/producers
70832025-07-17T11:36:50.682ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "37cee41b-7170-4a71-88b0-f56b8aa673aa", "content-length": "41", "date": "Thu, 17 Jul 2025 11:36:50 GMT"} })
70842025-07-17T11:36:50.684ZDEBGnexus (producer-registration-task): successfully registered with Nexus, pausing until time to renew lease lease_duration = 600s wait_period = 150s
70852025-07-17T11:36:50.758ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "d4a50d7c-2ccd-439a-9a20-2d26a8b2deb6", "date": "Thu, 17 Jul 2025 11:36:49 GMT"} })
70862025-07-17T11:36:50.758ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch = Switch0
70872025-07-17T11:36:50.758ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
70882025-07-17T11:36:50.759ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "66d60db4-5e3a-4592-94e8-77d7f600988e", "content-length": "682", "date": "Thu, 17 Jul 2025 11:36:50 GMT"} })
70892025-07-17T11:36:50.759ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
70902025-07-17T11:36:50.759ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
70912025-07-17T11:36:50.763ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
70922025-07-17T11:36:50.763ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
70932025-07-17T11:36:50.767ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
70942025-07-17T11:36:50.768ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
70952025-07-17T11:36:50.770ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
70962025-07-17T11:36:50.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
70972025-07-17T11:36:50.778ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 1.013048875s iteration = 23 status = {}
70982025-07-17T11:36:51.694ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 325 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = 327c3883-1134-4968-a538-3bbe19fc22e7 response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
70992025-07-17T11:36:54.765ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 134 reason = Timeout
71002025-07-17T11:36:54.765ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
71012025-07-17T11:36:54.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
71022025-07-17T11:36:54.767ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71032025-07-17T11:36:54.770ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
71042025-07-17T11:36:54.770ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
71052025-07-17T11:36:54.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
71062025-07-17T11:36:54.772ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
71072025-07-17T11:36:54.772ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 7.150284ms iteration = 134 status = {"chicken_switches_updated":false}
71082025-07-17T11:36:59.765ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 68 reason = Timeout
71092025-07-17T11:36:59.765ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
71102025-07-17T11:36:59.765ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 68 reason = Timeout
71112025-07-17T11:36:59.765ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 135 reason = Timeout
71122025-07-17T11:36:59.765ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
71132025-07-17T11:36:59.765ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
71142025-07-17T11:36:59.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
71152025-07-17T11:36:59.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
71162025-07-17T11:36:59.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
71172025-07-17T11:36:59.767ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
71182025-07-17T11:36:59.767ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
71192025-07-17T11:36:59.768ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71202025-07-17T11:36:59.768ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71212025-07-17T11:36:59.769ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.416929ms iteration = 68 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
71222025-07-17T11:36:59.771ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
71232025-07-17T11:36:59.771ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
71242025-07-17T11:36:59.771ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
71252025-07-17T11:36:59.772ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
71262025-07-17T11:36:59.772ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
71272025-07-17T11:36:59.772ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
71282025-07-17T11:36:59.773ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
71292025-07-17T11:36:59.773ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 8.604993ms iteration = 135 status = {"chicken_switches_updated":false}
71302025-07-17T11:36:59.774ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
71312025-07-17T11:36:59.774ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
71322025-07-17T11:36:59.776ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71332025-07-17T11:36:59.778ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
71342025-07-17T11:36:59.778ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
71352025-07-17T11:36:59.778ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
71362025-07-17T11:36:59.790ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
71372025-07-17T11:36:59.790ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 25.278783ms iteration = 68 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
71382025-07-17T11:37:01.693ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 367 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = c2885c20-6f4c-4476-a430-cf2fdcfda6c6 response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
71392025-07-17T11:37:04.765ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 136 reason = Timeout
71402025-07-17T11:37:04.765ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
71412025-07-17T11:37:04.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
71422025-07-17T11:37:04.767ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71432025-07-17T11:37:04.770ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
71442025-07-17T11:37:04.770ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
71452025-07-17T11:37:04.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
71462025-07-17T11:37:04.771ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
71472025-07-17T11:37:04.771ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 6.279243ms iteration = 136 status = {"chicken_switches_updated":false}
71482025-07-17T11:37:09.765ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 69 reason = Timeout
71492025-07-17T11:37:09.765ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
71502025-07-17T11:37:09.765ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 69 reason = Timeout
71512025-07-17T11:37:09.765ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 137 reason = Timeout
71522025-07-17T11:37:09.765ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
71532025-07-17T11:37:09.765ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
71542025-07-17T11:37:09.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
71552025-07-17T11:37:09.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
71562025-07-17T11:37:09.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
71572025-07-17T11:37:09.767ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
71582025-07-17T11:37:09.767ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
71592025-07-17T11:37:09.767ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71602025-07-17T11:37:09.768ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71612025-07-17T11:37:09.769ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 3.924262ms iteration = 69 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
71622025-07-17T11:37:09.771ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
71632025-07-17T11:37:09.771ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
71642025-07-17T11:37:09.771ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
71652025-07-17T11:37:09.771ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
71662025-07-17T11:37:09.771ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
71672025-07-17T11:37:09.772ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
71682025-07-17T11:37:09.772ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
71692025-07-17T11:37:09.772ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 7.427708ms iteration = 137 status = {"chicken_switches_updated":false}
71702025-07-17T11:37:09.773ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
71712025-07-17T11:37:09.774ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
71722025-07-17T11:37:09.775ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71732025-07-17T11:37:09.778ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
71742025-07-17T11:37:09.778ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
71752025-07-17T11:37:09.778ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
71762025-07-17T11:37:09.787ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
71772025-07-17T11:37:09.787ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.795951ms iteration = 69 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
71782025-07-17T11:37:11.694ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 558 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = 9aec981b-9fb7-481f-b8e8-4e2f792a5174 response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
71792025-07-17T11:37:14.766ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 138 reason = Timeout
71802025-07-17T11:37:14.766ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
71812025-07-17T11:37:14.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
71822025-07-17T11:37:14.768ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71832025-07-17T11:37:14.770ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
71842025-07-17T11:37:14.770ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
71852025-07-17T11:37:14.771ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
71862025-07-17T11:37:14.772ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
71872025-07-17T11:37:14.772ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 6.06645ms iteration = 138 status = {"chicken_switches_updated":false}
71882025-07-17T11:37:19.677ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 24 reason = Timeout
71892025-07-17T11:37:19.677ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
71902025-07-17T11:37:19.677ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
71912025-07-17T11:37:19.679ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 4033, tv_nsec: 430638206 } })
71922025-07-17T11:37:19.679ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1186
71932025-07-17T11:37:19.709ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1194 zone_address = fd00:1122:3344:101::2
71942025-07-17T11:37:19.709ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
71952025-07-17T11:37:19.710ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "baac6f8f-8f5e-4d05-909d-de90a375dfb4", "content-length": "26", "date": "Thu, 17 Jul 2025 11:37:19 GMT"} })
71962025-07-17T11:37:19.710ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1197 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
71972025-07-17T11:37:19.710ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
71982025-07-17T11:37:19.734ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 24 reason = Timeout
71992025-07-17T11:37:19.734ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 24 reason = Timeout
72002025-07-17T11:37:19.734ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
72012025-07-17T11:37:19.735ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
72022025-07-17T11:37:19.738ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 4033, tv_nsec: 489320570 } })
72032025-07-17T11:37:19.738ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.644368ms iteration = 24 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
72042025-07-17T11:37:19.738ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1186
72052025-07-17T11:37:19.742ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
72062025-07-17T11:37:19.743ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
72072025-07-17T11:37:19.745ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 68.441452ms iteration = 24 status = null
72082025-07-17T11:37:19.765ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 24 reason = Timeout
72092025-07-17T11:37:19.765ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
72102025-07-17T11:37:19.765ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 24 reason = Timeout
72112025-07-17T11:37:19.765ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
72122025-07-17T11:37:19.765ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
72132025-07-17T11:37:19.765ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 24 reason = Timeout
72142025-07-17T11:37:19.765ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
72152025-07-17T11:37:19.765ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 24 reason = Timeout
72162025-07-17T11:37:19.765ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
72172025-07-17T11:37:19.765ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 24 reason = Timeout
72182025-07-17T11:37:19.765ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
72192025-07-17T11:37:19.766ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 24 reason = Timeout
72202025-07-17T11:37:19.766ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
72212025-07-17T11:37:19.766ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 26 reason = Timeout
72222025-07-17T11:37:19.766ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
72232025-07-17T11:37:19.766ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 24 reason = Timeout
72242025-07-17T11:37:19.766ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
72252025-07-17T11:37:19.766ZDEBGnexus (BackgroundTasks): activating background_task = sp_ereport_ingester iteration = 24 reason = Timeout
72262025-07-17T11:37:19.766ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
72272025-07-17T11:37:19.766ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 24 reason = Timeout
72282025-07-17T11:37:19.766ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
72292025-07-17T11:37:19.766ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 24 reason = Timeout
72302025-07-17T11:37:19.766ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
72312025-07-17T11:37:19.766ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 24 reason = Timeout
72322025-07-17T11:37:19.766ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
72332025-07-17T11:37:19.766ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 24 reason = Timeout
72342025-07-17T11:37:19.766ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
72352025-07-17T11:37:19.766ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 24 reason = Timeout
72362025-07-17T11:37:19.766ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 24 reason = Timeout
72372025-07-17T11:37:19.766ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
72382025-07-17T11:37:19.766ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 70 reason = Timeout
72392025-07-17T11:37:19.766ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
72402025-07-17T11:37:19.766ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
72412025-07-17T11:37:19.766ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 70 reason = Timeout
72422025-07-17T11:37:19.766ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
72432025-07-17T11:37:19.766ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 139 reason = Timeout
72442025-07-17T11:37:19.766ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
72452025-07-17T11:37:19.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
72462025-07-17T11:37:19.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
72472025-07-17T11:37:19.766ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12225, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 4033, tv_nsec: 517699134 } })
72482025-07-17T11:37:19.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
72492025-07-17T11:37:19.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
72502025-07-17T11:37:19.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
72512025-07-17T11:37:19.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
72522025-07-17T11:37:19.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
72532025-07-17T11:37:19.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
72542025-07-17T11:37:19.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
72552025-07-17T11:37:19.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
72562025-07-17T11:37:19.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
72572025-07-17T11:37:19.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
72582025-07-17T11:37:19.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
72592025-07-17T11:37:19.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
72602025-07-17T11:37:19.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
72612025-07-17T11:37:19.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
72622025-07-17T11:37:19.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
72632025-07-17T11:37:19.768ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 2.175499ms iteration = 24 status = {"errors":[],"finish_invoked_ok":[]}
72642025-07-17T11:37:19.769ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72652025-07-17T11:37:19.770ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72662025-07-17T11:37:19.770ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72672025-07-17T11:37:19.771ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
72682025-07-17T11:37:19.771ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
72692025-07-17T11:37:19.771ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
72702025-07-17T11:37:19.771ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
72712025-07-17T11:37:19.772ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72722025-07-17T11:37:19.772ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
72732025-07-17T11:37:19.772ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
72742025-07-17T11:37:19.772ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1194 zone_address = fd00:1122:3344:101::2
72752025-07-17T11:37:19.772ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
72762025-07-17T11:37:19.772ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
72772025-07-17T11:37:19.772ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "e3706e88-99fb-48a5-9de1-4ad3d6fcec1a", "content-length": "26", "date": "Thu, 17 Jul 2025 11:37:19 GMT"} })
72782025-07-17T11:37:19.772ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1197 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
72792025-07-17T11:37:19.773ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
72802025-07-17T11:37:19.773ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
72812025-07-17T11:37:19.773ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
72822025-07-17T11:37:19.774ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 8.030436ms iteration = 24 status = {"errors":[],"requests_created_ok":[]}
72832025-07-17T11:37:19.774ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72842025-07-17T11:37:19.774ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
72852025-07-17T11:37:19.774ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
72862025-07-17T11:37:19.774ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
72872025-07-17T11:37:19.774ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
72882025-07-17T11:37:19.774ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
72892025-07-17T11:37:19.774ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
72902025-07-17T11:37:19.774ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
72912025-07-17T11:37:19.775ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72922025-07-17T11:37:19.776ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
72932025-07-17T11:37:19.776ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
72942025-07-17T11:37:19.776ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
72952025-07-17T11:37:19.776ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
72962025-07-17T11:37:19.776ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
72972025-07-17T11:37:19.776ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
72982025-07-17T11:37:19.776ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
72992025-07-17T11:37:19.777ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
73002025-07-17T11:37:19.777ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
73012025-07-17T11:37:19.777ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
73022025-07-17T11:37:19.777ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
73032025-07-17T11:37:19.777ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
73042025-07-17T11:37:19.777ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
73052025-07-17T11:37:19.778ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
73062025-07-17T11:37:19.778ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73072025-07-17T11:37:19.779ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 4033, tv_nsec: 530773516 } })
73082025-07-17T11:37:19.779ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1186 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
73092025-07-17T11:37:19.780ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
73102025-07-17T11:37:19.780ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
73112025-07-17T11:37:19.781ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
73122025-07-17T11:37:19.823ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73132025-07-17T11:37:19.823ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
73142025-07-17T11:37:19.823ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
73152025-07-17T11:37:19.824ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73162025-07-17T11:37:19.827ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
73172025-07-17T11:37:19.827ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
73182025-07-17T11:37:19.827ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
73192025-07-17T11:37:19.830ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
73202025-07-17T11:37:19.830ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
73212025-07-17T11:37:19.830ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
73222025-07-17T11:37:19.832ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
73232025-07-17T11:37:19.832ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 66.064581ms iteration = 139 status = {"chicken_switches_updated":false}
73242025-07-17T11:37:19.832ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
73252025-07-17T11:37:19.833ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
73262025-07-17T11:37:19.833ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73272025-07-17T11:37:19.838ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
73282025-07-17T11:37:19.838ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
73292025-07-17T11:37:19.838ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
73302025-07-17T11:37:19.842ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
73312025-07-17T11:37:19.842ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
73322025-07-17T11:37:19.843ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
73332025-07-17T11:37:19.843ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
73342025-07-17T11:37:19.844ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 78.8984ms iteration = 24 status = {"errors":[],"garbage_collect_requested":[]}
73352025-07-17T11:37:19.845ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73362025-07-17T11:37:19.849ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
73372025-07-17T11:37:19.849ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
73382025-07-17T11:37:19.849ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
73392025-07-17T11:37:19.849ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
73402025-07-17T11:37:19.849ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
73412025-07-17T11:37:19.849ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
73422025-07-17T11:37:19.850ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
73432025-07-17T11:37:19.850ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
73442025-07-17T11:37:19.850ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
73452025-07-17T11:37:19.850ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 84.347521ms iteration = 24 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
73462025-07-17T11:37:19.854ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
73472025-07-17T11:37:19.854ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:550 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
73482025-07-17T11:37:19.854ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 88.682319ms iteration = 24 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
73492025-07-17T11:37:19.855ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
73502025-07-17T11:37:19.856ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
73512025-07-17T11:37:19.856ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73522025-07-17T11:37:19.857ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
73532025-07-17T11:37:19.859ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73542025-07-17T11:37:19.862ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 0ae78f24-48db-4cf7-91cb-8d5311f2a17f, lookup_type: ById(0ae78f24-48db-4cf7-91cb-8d5311f2a17f) } result = Ok(())
73552025-07-17T11:37:19.862ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
73562025-07-17T11:37:19.862ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
73572025-07-17T11:37:19.864ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
73582025-07-17T11:37:19.865ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
73592025-07-17T11:37:19.865ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
73602025-07-17T11:37:19.866ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
73612025-07-17T11:37:19.867ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
73622025-07-17T11:37:19.871ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 105.677944ms iteration = 24 status = 0
73632025-07-17T11:37:19.873ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 106.910739ms iteration = 24 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
73642025-07-17T11:37:19.888ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
73652025-07-17T11:37:19.888ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
73662025-07-17T11:37:19.891ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73672025-07-17T11:37:19.891ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
73682025-07-17T11:37:19.891ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
73692025-07-17T11:37:19.894ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 128.023868ms iteration = 24 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
73702025-07-17T11:37:19.896ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
73712025-07-17T11:37:19.896ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
73722025-07-17T11:37:19.897ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
73732025-07-17T11:37:19.900ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73742025-07-17T11:37:19.902ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
73752025-07-17T11:37:19.902ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
73762025-07-17T11:37:19.903ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
73772025-07-17T11:37:19.903ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
73782025-07-17T11:37:19.903ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 137.658325ms iteration = 70 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
73792025-07-17T11:37:19.923ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1194 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 zone_address = fd00:1122:3344:101::2
73802025-07-17T11:37:19.923ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
73812025-07-17T11:37:19.924ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "882e2756-5531-4f70-af86-81a3e4eeea97", "content-length": "26", "date": "Thu, 17 Jul 2025 11:37:19 GMT"} })
73822025-07-17T11:37:19.924ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1197 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
73832025-07-17T11:37:19.926ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
73842025-07-17T11:37:19.929ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 163.683458ms iteration = 24 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
73852025-07-17T11:37:19.930ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:372
73862025-07-17T11:37:19.930ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 164.060893ms iteration = 24 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
73872025-07-17T11:37:19.935ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 168.995058ms iteration = 70 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
73882025-07-17T11:37:19.951ZINFOnexus (BackgroundTasks): VPC route manager sled 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
73892025-07-17T11:37:19.951ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
73902025-07-17T11:37:19.952ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "19a7e36f-3b6d-4ee3-95f9-cd6d1c6657d3", "content-length": "372", "date": "Thu, 17 Jul 2025 11:37:19 GMT"} })
73912025-07-17T11:37:19.952ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
73922025-07-17T11:37:19.952ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
73932025-07-17T11:37:19.968ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/all-sp-ids
73942025-07-17T11:37:19.969ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/all-sp-ids", status: 200, headers: {"content-type": "application/json", "x-request-id": "91198ea2-01dc-4d68-b7c6-6e70b4fd3176", "content-length": "78", "date": "Thu, 17 Jul 2025 11:37:19 GMT"} })
73952025-07-17T11:37:19.969ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = switch
73962025-07-17T11:37:19.969ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = sled
73972025-07-17T11:37:19.969ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 1 sp_type = sled
73982025-07-17T11:37:19.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = sled
73992025-07-17T11:37:19.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = switch
74002025-07-17T11:37:19.969ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 1 sp_type = sled
74012025-07-17T11:37:19.969ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
74022025-07-17T11:37:19.969ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
74032025-07-17T11:37:19.970ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "87c6ec3e-8d3e-4a47-a4e2-40240fac2f36", "date": "Thu, 17 Jul 2025 11:37:19 GMT"} })
74042025-07-17T11:37:19.970ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
74052025-07-17T11:37:19.970ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
74062025-07-17T11:37:19.971ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 0 sp_type = switch
74072025-07-17T11:37:19.972ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 1 sp_type = sled
74082025-07-17T11:37:19.974ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 0 sp_type = switch
74092025-07-17T11:37:19.974ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = switch
74102025-07-17T11:37:19.974ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = switch
74112025-07-17T11:37:19.979ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 1 sp_type = sled
74122025-07-17T11:37:19.979ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
74132025-07-17T11:37:19.979ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 1 sp_type = sled
74142025-07-17T11:37:19.979ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 1 sp_type = sled
74152025-07-17T11:37:19.979ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
74162025-07-17T11:37:19.982ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 248.127522ms iteration = 24 status = {}
74172025-07-17T11:37:19.983ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 1 sp_type = sled start_ena = None
74182025-07-17T11:37:19.983ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/1/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
74192025-07-17T11:37:19.983ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/1/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "4aed6850-dae2-4e4e-83d3-e5ff60cbf104", "content-length": "81", "date": "Thu, 17 Jul 2025 11:37:19 GMT"} })
74202025-07-17T11:37:19.990ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
74212025-07-17T11:37:19.990ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
74222025-07-17T11:37:19.994ZINFOnexus (BackgroundTasks): VPC route manager sled 0ae78f24-48db-4cf7-91cb-8d5311f2a17f push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
74232025-07-17T11:37:19.994ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
74242025-07-17T11:37:19.995ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "35f6ee24-c08f-42d6-8a76-8ab1a868f59c", "date": "Thu, 17 Jul 2025 11:37:19 GMT"} })
74252025-07-17T11:37:19.996ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 230.952825ms iteration = 24 status = {}
74262025-07-17T11:37:20.009ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 243.588842ms iteration = 26 status = {"physical_disks_added":0}
74272025-07-17T11:37:20.027ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 0 sp_type = sled
74282025-07-17T11:37:20.028ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 0 sp_type = switch start_ena = None
74292025-07-17T11:37:20.028ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
74302025-07-17T11:37:20.029ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "25d22819-96e8-4957-be3c-3faf318b0d7a", "content-length": "81", "date": "Thu, 17 Jul 2025 11:37:19 GMT"} })
74312025-07-17T11:37:20.029ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 0 sp_type = sled
74322025-07-17T11:37:20.029ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = sled
74332025-07-17T11:37:20.029ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = sled
74342025-07-17T11:37:20.064ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
74352025-07-17T11:37:20.065ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "ddd5e7e1-4b41-404c-b490-88fd72180a12", "content-length": "691", "date": "Thu, 17 Jul 2025 11:37:19 GMT"} })
74362025-07-17T11:37:20.065ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
74372025-07-17T11:37:20.065ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
74382025-07-17T11:37:20.065ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
74392025-07-17T11:37:20.066ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 300.842226ms iteration = 24 status = {}
74402025-07-17T11:37:20.069ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 0 sp_type = sled start_ena = None
74412025-07-17T11:37:20.069ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
74422025-07-17T11:37:20.069ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "837e7d15-1bf4-41a0-8068-ebc5b9791d6f", "content-length": "81", "date": "Thu, 17 Jul 2025 11:37:19 GMT"} })
74432025-07-17T11:37:20.071ZDEBGnexus (BackgroundTasks): activation complete background_task = sp_ereport_ingester elapsed = 305.116543ms iteration = 24 status = {"errors":[],"sps":[]}
74442025-07-17T11:37:20.077ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
74452025-07-17T11:37:20.077ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
74462025-07-17T11:37:20.080ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 151caa7c-6a05-4995-94b4-7946d428117d, rack_id: 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5, switch_location: "switch0", port_name: Name(Name("qsfp0")), port_settings_id: Some(2e30c7ae-26da-46e4-9127-aac194921f20) } rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0
74472025-07-17T11:37:20.080ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
74482025-07-17T11:37:20.080ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
74492025-07-17T11:37:20.090ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1832 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = Name(Name("qsfp0"))
74502025-07-17T11:37:20.090ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1859 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0 switch_port_id = Name(Name("qsfp0")) switch_port_settings_id = 2e30c7ae-26da-46e4-9127-aac194921f20
74512025-07-17T11:37:20.091ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
74522025-07-17T11:37:20.091ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
74532025-07-17T11:37:20.091ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
74542025-07-17T11:37:20.091ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
74552025-07-17T11:37:20.091ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
74562025-07-17T11:37:20.091ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2006 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
74572025-07-17T11:37:20.091ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
74582025-07-17T11:37:20.091ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2042 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
74592025-07-17T11:37:20.091ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
74602025-07-17T11:37:20.091ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
74612025-07-17T11:37:20.092ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
74622025-07-17T11:37:20.094ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0
74632025-07-17T11:37:20.094ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
74642025-07-17T11:37:20.305ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "0000a625-3dc5-430d-ada1-24073795d7ba", "date": "Thu, 17 Jul 2025 11:37:19 GMT"} })
74652025-07-17T11:37:20.305ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch = Switch0
74662025-07-17T11:37:20.305ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
74672025-07-17T11:37:20.311ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "b41bf725-6028-4986-af5b-292076e2fd26", "content-length": "682", "date": "Thu, 17 Jul 2025 11:37:20 GMT"} })
74682025-07-17T11:37:20.311ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
74692025-07-17T11:37:20.311ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
74702025-07-17T11:37:20.315ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
74712025-07-17T11:37:20.315ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
74722025-07-17T11:37:20.319ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
74732025-07-17T11:37:20.319ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
74742025-07-17T11:37:20.321ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
74752025-07-17T11:37:20.322ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
74762025-07-17T11:37:20.329ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 563.749173ms iteration = 24 status = {}
74772025-07-17T11:37:21.694ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 344 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = fc341432-a936-4166-a398-ba41d60f4d8f response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
74782025-07-17T11:37:24.765ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 140 reason = Timeout
74792025-07-17T11:37:24.765ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
74802025-07-17T11:37:24.765ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
74812025-07-17T11:37:24.767ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74822025-07-17T11:37:24.770ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
74832025-07-17T11:37:24.770ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
74842025-07-17T11:37:24.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
74852025-07-17T11:37:24.771ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
74862025-07-17T11:37:24.771ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 6.377914ms iteration = 140 status = {"chicken_switches_updated":false}
74872025-07-17T11:37:29.765ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 71 reason = Timeout
74882025-07-17T11:37:29.765ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 71 reason = Timeout
74892025-07-17T11:37:29.765ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
74902025-07-17T11:37:29.765ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
74912025-07-17T11:37:29.765ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 141 reason = Timeout
74922025-07-17T11:37:29.765ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
74932025-07-17T11:37:29.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
74942025-07-17T11:37:29.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
74952025-07-17T11:37:29.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
74962025-07-17T11:37:29.768ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
74972025-07-17T11:37:29.768ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
74982025-07-17T11:37:29.768ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74992025-07-17T11:37:29.768ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75002025-07-17T11:37:29.769ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.194715ms iteration = 71 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
75012025-07-17T11:37:29.771ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
75022025-07-17T11:37:29.771ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
75032025-07-17T11:37:29.771ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
75042025-07-17T11:37:29.772ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
75052025-07-17T11:37:29.772ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
75062025-07-17T11:37:29.772ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
75072025-07-17T11:37:29.774ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
75082025-07-17T11:37:29.774ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
75092025-07-17T11:37:29.774ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
75102025-07-17T11:37:29.774ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 8.515093ms iteration = 141 status = {"chicken_switches_updated":false}
75112025-07-17T11:37:29.775ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75122025-07-17T11:37:29.778ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
75132025-07-17T11:37:29.778ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
75142025-07-17T11:37:29.778ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
75152025-07-17T11:37:29.788ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
75162025-07-17T11:37:29.788ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.554987ms iteration = 71 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
75172025-07-17T11:37:31.694ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 338 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = 25c0b5a2-9cac-4078-b915-81a3d742dad7 response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
75182025-07-17T11:37:34.765ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 142 reason = Timeout
75192025-07-17T11:37:34.765ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
75202025-07-17T11:37:34.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
75212025-07-17T11:37:34.767ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75222025-07-17T11:37:34.770ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
75232025-07-17T11:37:34.770ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
75242025-07-17T11:37:34.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
75252025-07-17T11:37:34.771ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
75262025-07-17T11:37:34.771ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 5.969488ms iteration = 142 status = {"chicken_switches_updated":false}
75272025-07-17T11:37:39.766ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 72 reason = Timeout
75282025-07-17T11:37:39.766ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
75292025-07-17T11:37:39.766ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 72 reason = Timeout
75302025-07-17T11:37:39.766ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 143 reason = Timeout
75312025-07-17T11:37:39.766ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
75322025-07-17T11:37:39.766ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
75332025-07-17T11:37:39.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
75342025-07-17T11:37:39.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
75352025-07-17T11:37:39.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
75362025-07-17T11:37:39.768ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
75372025-07-17T11:37:39.768ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
75382025-07-17T11:37:39.769ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75392025-07-17T11:37:39.769ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75402025-07-17T11:37:39.770ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.477129ms iteration = 72 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
75412025-07-17T11:37:39.772ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
75422025-07-17T11:37:39.772ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
75432025-07-17T11:37:39.772ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
75442025-07-17T11:37:39.773ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
75452025-07-17T11:37:39.773ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
75462025-07-17T11:37:39.773ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
75472025-07-17T11:37:39.774ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
75482025-07-17T11:37:39.774ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 8.005686ms iteration = 143 status = {"chicken_switches_updated":false}
75492025-07-17T11:37:39.775ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
75502025-07-17T11:37:39.775ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
75512025-07-17T11:37:39.776ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75522025-07-17T11:37:39.779ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
75532025-07-17T11:37:39.779ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
75542025-07-17T11:37:39.779ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
75552025-07-17T11:37:39.789ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
75562025-07-17T11:37:39.789ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 23.577981ms iteration = 72 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
75572025-07-17T11:37:41.694ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 391 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = d6727b91-1169-406d-937f-940c7e140e4f response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
75582025-07-17T11:37:44.765ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 144 reason = Timeout
75592025-07-17T11:37:44.765ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
75602025-07-17T11:37:44.766ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
75612025-07-17T11:37:44.767ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75622025-07-17T11:37:44.770ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
75632025-07-17T11:37:44.770ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
75642025-07-17T11:37:44.770ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
75652025-07-17T11:37:44.771ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
75662025-07-17T11:37:44.771ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 6.056049ms iteration = 144 status = {"chicken_switches_updated":false}
75672025-07-17T11:37:49.678ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 13 reason = Timeout
75682025-07-17T11:37:49.678ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 13 reason = Timeout
75692025-07-17T11:37:49.678ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
75702025-07-17T11:37:49.678ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 13 reason = Timeout
75712025-07-17T11:37:49.678ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 13 reason = Timeout
75722025-07-17T11:37:49.678ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-07-17 11:27:49.678321446 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
75732025-07-17T11:37:49.678ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 15 reason = Timeout
75742025-07-17T11:37:49.678ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
75752025-07-17T11:37:49.678ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 13 reason = Timeout
75762025-07-17T11:37:49.678ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
75772025-07-17T11:37:49.678ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
75782025-07-17T11:37:49.678ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 13 reason = Timeout
75792025-07-17T11:37:49.678ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
75802025-07-17T11:37:49.678ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
75812025-07-17T11:37:49.678ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 25 reason = Timeout
75822025-07-17T11:37:49.678ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 15 reason = Timeout
75832025-07-17T11:37:49.678ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
75842025-07-17T11:37:49.678ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
75852025-07-17T11:37:49.678ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
75862025-07-17T11:37:49.678ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
75872025-07-17T11:37:49.678ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
75882025-07-17T11:37:49.678ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
75892025-07-17T11:37:49.678ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("3144a75a-317f-4b68-93e2-a210ca1189d1.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("87408888-9df4-4e0c-bc8e-2f920d039100.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("ee4378c5-e5f2-41cf-8969-e5e03e1b9a41.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("ee4378c5-e5f2-41cf-8969-e5e03e1b9a41.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:3::1)) }], valid_until: Instant { tv_sec: 4063, tv_nsec: 429629959 } })
75902025-07-17T11:37:49.679ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("3dd3bf3b-7760-4750-b1d3-52ee777335f0.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("79e5c77d-18b3-4951-8d11-cfa3eae2b3e3.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("79e5c77d-18b3-4951-8d11-cfa3eae2b3e3.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::8)) }], valid_until: Instant { tv_sec: 4063, tv_nsec: 429637868 } })
75912025-07-17T11:37:49.679ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
75922025-07-17T11:37:49.679ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.488599ms iteration = 13 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
75932025-07-17T11:37:49.679ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } dns_group = internal
75942025-07-17T11:37:49.679ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.47827ms iteration = 13 status = {"addresses":["[fd00:1122:3344:1::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:3::1]:5353"]}
75952025-07-17T11:37:49.680ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 14 reason = Timeout
75962025-07-17T11:37:49.680ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:85 reason = blueprint disabled target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a
75972025-07-17T11:37:49.680ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 23.6\u{b5}s iteration = 14 status = {"enabled":false,"target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a"}
75982025-07-17T11:37:49.681ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75992025-07-17T11:37:49.681ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
76002025-07-17T11:37:49.681ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
76012025-07-17T11:37:49.683ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
76022025-07-17T11:37:49.683ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
76032025-07-17T11:37:49.684ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 4063, tv_nsec: 435047880 } })
76042025-07-17T11:37:49.684ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1186
76052025-07-17T11:37:49.685ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-07-17 11:27:49.678321446 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
76062025-07-17T11:37:49.685ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 6.689768ms iteration = 13 status = {"expiration":"2025-07-17T11:27:49.678321446Z","pruned":[]}
76072025-07-17T11:37:49.685ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
76082025-07-17T11:37:49.685ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
76092025-07-17T11:37:49.685ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
76102025-07-17T11:37:49.686ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76112025-07-17T11:37:49.686ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76122025-07-17T11:37:49.686ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_planner iteration = 16 reason = Timeout
76132025-07-17T11:37:49.686ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 13 reason = Timeout
76142025-07-17T11:37:49.686ZDEBGnexus (BackgroundTasks): blueprint planning disabled, doing nothing background_task = blueprint_planner
76152025-07-17T11:37:49.686ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:53
76162025-07-17T11:37:49.686ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_planner elapsed = 10.34\u{b5}s iteration = 16 status = "Disabled"
76172025-07-17T11:37:49.686ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 13 reason = Timeout
76182025-07-17T11:37:49.686ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
76192025-07-17T11:37:49.686ZDEBGnexus (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 6 reason = Timeout
76202025-07-17T11:37:49.686ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
76212025-07-17T11:37:49.686ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
76222025-07-17T11:37:49.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
76232025-07-17T11:37:49.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
76242025-07-17T11:37:49.687ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
76252025-07-17T11:37:49.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
76262025-07-17T11:37:49.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
76272025-07-17T11:37:49.687ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
76282025-07-17T11:37:49.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
76292025-07-17T11:37:49.687ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
76302025-07-17T11:37:49.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
76312025-07-17T11:37:49.687ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
76322025-07-17T11:37:49.688ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
76332025-07-17T11:37:49.688ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 13 reason = Timeout
76342025-07-17T11:37:49.688ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
76352025-07-17T11:37:49.688ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
76362025-07-17T11:37:49.688ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
76372025-07-17T11:37:49.689ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
76382025-07-17T11:37:49.689ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76392025-07-17T11:37:49.690ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76402025-07-17T11:37:49.693ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
76412025-07-17T11:37:49.693ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
76422025-07-17T11:37:49.693ZDEBGnexus (BackgroundTasks): activating background_task = webhook_deliverator iteration = 13 reason = Timeout
76432025-07-17T11:37:49.693ZDEBGnexus (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} }
76442025-07-17T11:37:49.693ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 13 reason = Timeout
76452025-07-17T11:37:49.693ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
76462025-07-17T11:37:49.693ZDEBGnexus (BackgroundTasks): activating background_task = alert_dispatcher iteration = 13 reason = Timeout
76472025-07-17T11:37:49.693ZDEBGnexus (BackgroundTasks): roles background_task = alert_dispatcher roles = RoleSet { roles: {} }
76482025-07-17T11:37:49.694ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
76492025-07-17T11:37:49.694ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
76502025-07-17T11:37:49.694ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database result = Ok(())
76512025-07-17T11:37:49.694ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
76522025-07-17T11:37:49.694ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
76532025-07-17T11:37:49.694ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76542025-07-17T11:37:49.694ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
76552025-07-17T11:37:49.694ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
76562025-07-17T11:37:49.694ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
76572025-07-17T11:37:49.694ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
76582025-07-17T11:37:49.694ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = alert_dispatcher resource = Database result = Ok(())
76592025-07-17T11:37:49.694ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
76602025-07-17T11:37:49.694ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
76612025-07-17T11:37:49.694ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
76622025-07-17T11:37:49.695ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
76632025-07-17T11:37:49.695ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
76642025-07-17T11:37:49.695ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
76652025-07-17T11:37:49.695ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
76662025-07-17T11:37:49.695ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
76672025-07-17T11:37:49.695ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
76682025-07-17T11:37:49.697ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
76692025-07-17T11:37:49.697ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
76702025-07-17T11:37:49.697ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
76712025-07-17T11:37:49.700ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76722025-07-17T11:37:49.700ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76732025-07-17T11:37:49.702ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 8.468482ms iteration = 13 status = {"errors":[],"found_port_ok":[]}
76742025-07-17T11:37:49.703ZINFOnexus (BackgroundTasks): all webhook delivery tasks completed background_task = webhook_deliverator file = nexus/src/app/background/tasks/webhook_deliverator.rs:204 num_receivers = 0
76752025-07-17T11:37:49.703ZDEBGnexus (BackgroundTasks): activation complete background_task = webhook_deliverator elapsed = 9.227852ms iteration = 13 status = {"by_rx":{},"error":null}
76762025-07-17T11:37:49.704ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
76772025-07-17T11:37:49.704ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
76782025-07-17T11:37:49.704ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
76792025-07-17T11:37:49.704ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76802025-07-17T11:37:49.706ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76812025-07-17T11:37:49.706ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
76822025-07-17T11:37:49.706ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
76832025-07-17T11:37:49.707ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
76842025-07-17T11:37:49.708ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76852025-07-17T11:37:49.708ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
76862025-07-17T11:37:49.708ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
76872025-07-17T11:37:49.708ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
76882025-07-17T11:37:49.708ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76892025-07-17T11:37:49.712ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
76902025-07-17T11:37:49.712ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
76912025-07-17T11:37:49.712ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
76922025-07-17T11:37:49.712ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Fleet result = Ok(())
76932025-07-17T11:37:49.712ZDEBGnexus (BackgroundTasks): roles background_task = crdb_node_id_collector roles = RoleSet { roles: {} }
76942025-07-17T11:37:49.712ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = crdb_node_id_collector resource = Database result = Ok(())
76952025-07-17T11:37:49.713ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
76962025-07-17T11:37:49.713ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
76972025-07-17T11:37:49.713ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
76982025-07-17T11:37:49.722ZDEBGnexus (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 35.566489ms iteration = 6 status = {"nsuccess":5}
76992025-07-17T11:37:49.723ZDEBGnexus (BackgroundTasks): roles background_task = alert_dispatcher roles = RoleSet { roles: {} }
77002025-07-17T11:37:49.724ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = alert_dispatcher resource = Database result = Ok(())
77012025-07-17T11:37:49.727ZDEBGnexus (BackgroundTasks): activation complete background_task = alert_dispatcher elapsed = 33.804935ms iteration = 13 status = {"dispatched":[],"errors":[],"glob_version":"161.0.0","globs_reprocessed":{},"no_receivers":[]}
77022025-07-17T11:37:49.735ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 25 reason = Timeout
77032025-07-17T11:37:49.735ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 25 reason = Timeout
77042025-07-17T11:37:49.735ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
77052025-07-17T11:37:49.735ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
77062025-07-17T11:37:49.745ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 10.006063ms iteration = 25 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
77072025-07-17T11:37:49.746ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 4063, tv_nsec: 496755833 } })
77082025-07-17T11:37:49.746ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1186
77092025-07-17T11:37:49.781ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
77102025-07-17T11:37:49.781ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:170 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
77112025-07-17T11:37:49.781ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 25 reason = Timeout
77122025-07-17T11:37:49.781ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 93.070397ms iteration = 13 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
77132025-07-17T11:37:49.781ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
77142025-07-17T11:37:49.781ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 25 reason = Timeout
77152025-07-17T11:37:49.781ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
77162025-07-17T11:37:49.781ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 25 reason = Timeout
77172025-07-17T11:37:49.781ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 27 reason = Timeout
77182025-07-17T11:37:49.781ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 25 reason = Timeout
77192025-07-17T11:37:49.781ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
77202025-07-17T11:37:49.781ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
77212025-07-17T11:37:49.781ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
77222025-07-17T11:37:49.781ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 25 reason = Timeout
77232025-07-17T11:37:49.781ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 25 reason = Timeout
77242025-07-17T11:37:49.781ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
77252025-07-17T11:37:49.781ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
77262025-07-17T11:37:49.781ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 25 reason = Timeout
77272025-07-17T11:37:49.781ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
77282025-07-17T11:37:49.781ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
77292025-07-17T11:37:49.781ZDEBGnexus (BackgroundTasks): activating background_task = sp_ereport_ingester iteration = 25 reason = Timeout
77302025-07-17T11:37:49.781ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 73 reason = Timeout
77312025-07-17T11:37:49.781ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 25 reason = Timeout
77322025-07-17T11:37:49.781ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
77332025-07-17T11:37:49.781ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
77342025-07-17T11:37:49.781ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 25 reason = Timeout
77352025-07-17T11:37:49.781ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
77362025-07-17T11:37:49.781ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 25 reason = Timeout
77372025-07-17T11:37:49.781ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
77382025-07-17T11:37:49.781ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 145 reason = Timeout
77392025-07-17T11:37:49.781ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
77402025-07-17T11:37:49.781ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 25 reason = Timeout
77412025-07-17T11:37:49.781ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
77422025-07-17T11:37:49.781ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 73 reason = Timeout
77432025-07-17T11:37:49.781ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
77442025-07-17T11:37:49.781ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 25 reason = Timeout
77452025-07-17T11:37:49.781ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
77462025-07-17T11:37:49.781ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 25 reason = Timeout
77472025-07-17T11:37:49.781ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
77482025-07-17T11:37:49.781ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
77492025-07-17T11:37:49.781ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
77502025-07-17T11:37:49.781ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
77512025-07-17T11:37:49.781ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
77522025-07-17T11:37:49.781ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
77532025-07-17T11:37:49.781ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
77542025-07-17T11:37:49.781ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
77552025-07-17T11:37:49.781ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
77562025-07-17T11:37:49.781ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
77572025-07-17T11:37:49.781ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12225, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 4063, tv_nsec: 532640337 } })
77582025-07-17T11:37:49.781ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
77592025-07-17T11:37:49.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
77602025-07-17T11:37:49.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
77612025-07-17T11:37:49.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
77622025-07-17T11:37:49.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
77632025-07-17T11:37:49.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
77642025-07-17T11:37:49.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
77652025-07-17T11:37:49.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
77662025-07-17T11:37:49.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
77672025-07-17T11:37:49.784ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77682025-07-17T11:37:49.784ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 3.702688ms iteration = 25 status = {"errors":[],"garbage_collect_requested":[]}
77692025-07-17T11:37:49.787ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
77702025-07-17T11:37:49.787ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
77712025-07-17T11:37:49.788ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
77722025-07-17T11:37:49.788ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
77732025-07-17T11:37:49.788ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 915fbba0-947f-4bf0-9f9f-48a79df81f7b dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
77742025-07-17T11:37:49.788ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
77752025-07-17T11:37:49.788ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77762025-07-17T11:37:49.788ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
77772025-07-17T11:37:49.790ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
77782025-07-17T11:37:49.790ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
77792025-07-17T11:37:49.790ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77802025-07-17T11:37:49.792ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
77812025-07-17T11:37:49.793ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
77822025-07-17T11:37:49.793ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
77832025-07-17T11:37:49.793ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
77842025-07-17T11:37:49.793ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
77852025-07-17T11:37:49.794ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
77862025-07-17T11:37:49.794ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
77872025-07-17T11:37:49.794ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
77882025-07-17T11:37:49.794ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
77892025-07-17T11:37:49.795ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77902025-07-17T11:37:49.795ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
77912025-07-17T11:37:49.795ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:138 found = 0 not_ready_to_be_deleted = 0
77922025-07-17T11:37:49.795ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 109.014637ms iteration = 13 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
77932025-07-17T11:37:49.795ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
77942025-07-17T11:37:49.795ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
77952025-07-17T11:37:49.796ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
77962025-07-17T11:37:49.796ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
77972025-07-17T11:37:49.796ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
77982025-07-17T11:37:49.796ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
77992025-07-17T11:37:49.797ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 4a9e4c5e-728e-4850-bd58-14548c02bf6f dns_zone_name = oxide.test found_so_far = 0 version = 2
78002025-07-17T11:37:49.797ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
78012025-07-17T11:37:49.797ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
78022025-07-17T11:37:49.797ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
78032025-07-17T11:37:49.797ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 16.332096ms iteration = 145 status = {"chicken_switches_updated":false}
78042025-07-17T11:37:49.797ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
78052025-07-17T11:37:49.798ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
78062025-07-17T11:37:49.798ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
78072025-07-17T11:37:49.799ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
78082025-07-17T11:37:49.799ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 18.353982ms iteration = 25 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
78092025-07-17T11:37:49.800ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
78102025-07-17T11:37:49.800ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
78112025-07-17T11:37:49.800ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78122025-07-17T11:37:49.801ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
78132025-07-17T11:37:49.801ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
78142025-07-17T11:37:49.801ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
78152025-07-17T11:37:49.801ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
78162025-07-17T11:37:49.801ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
78172025-07-17T11:37:49.801ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
78182025-07-17T11:37:49.802ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
78192025-07-17T11:37:49.802ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 20.898675ms iteration = 25 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
78202025-07-17T11:37:49.802ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78212025-07-17T11:37:49.804ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78222025-07-17T11:37:49.805ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
78232025-07-17T11:37:49.806ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
78242025-07-17T11:37:49.806ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
78252025-07-17T11:37:49.806ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
78262025-07-17T11:37:49.807ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
78272025-07-17T11:37:49.807ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 915fbba0-947f-4bf0-9f9f-48a79df81f7b dns_zone_name = control-plane.oxide.internal found_so_far = 59 version = 1
78282025-07-17T11:37:49.807ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
78292025-07-17T11:37:49.807ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
78302025-07-17T11:37:49.807ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
78312025-07-17T11:37:49.807ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-07-17 11:26:34.922237 UTC dns_group = internal generation = 1
78322025-07-17T11:37:49.807ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 128.798968ms iteration = 13 status = {"generation":1}
78332025-07-17T11:37:49.807ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
78342025-07-17T11:37:49.810ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
78352025-07-17T11:37:49.810ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
78362025-07-17T11:37:49.810ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
78372025-07-17T11:37:49.846ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 4a9e4c5e-728e-4850-bd58-14548c02bf6f dns_zone_name = oxide.test found_so_far = 4 version = 2
78382025-07-17T11:37:49.846ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
78392025-07-17T11:37:49.846ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-07-17 11:26:35.847594 UTC dns_group = external generation = 2
78402025-07-17T11:37:49.846ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 167.906624ms iteration = 13 status = {"generation":2}
78412025-07-17T11:37:49.848ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
78422025-07-17T11:37:49.851ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
78432025-07-17T11:37:49.852ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
78442025-07-17T11:37:49.854ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78452025-07-17T11:37:49.858ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
78462025-07-17T11:37:49.858ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
78472025-07-17T11:37:49.858ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
78482025-07-17T11:37:49.861ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78492025-07-17T11:37:49.865ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
78502025-07-17T11:37:49.865ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
78512025-07-17T11:37:49.866ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
78522025-07-17T11:37:49.880ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
78532025-07-17T11:37:49.880ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 99.404861ms iteration = 73 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
78542025-07-17T11:37:49.885ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78552025-07-17T11:37:49.885ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 104.344755ms iteration = 25 status = {"errors":[],"requests_created_ok":[]}
78562025-07-17T11:37:49.885ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 104.386766ms iteration = 73 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
78572025-07-17T11:37:49.888ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
78582025-07-17T11:37:49.889ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
78592025-07-17T11:37:49.889ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
78602025-07-17T11:37:49.889ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
78612025-07-17T11:37:49.890ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
78622025-07-17T11:37:49.890ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
78632025-07-17T11:37:49.893ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78642025-07-17T11:37:49.898ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
78652025-07-17T11:37:49.898ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
78662025-07-17T11:37:49.898ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
78672025-07-17T11:37:49.914ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 132.991353ms iteration = 25 status = {"errors":[],"finish_invoked_ok":[]}
78682025-07-17T11:37:49.914ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78692025-07-17T11:37:49.917ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
78702025-07-17T11:37:49.917ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
78712025-07-17T11:37:49.918ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
78722025-07-17T11:37:49.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
78732025-07-17T11:37:49.918ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
78742025-07-17T11:37:49.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
78752025-07-17T11:37:49.918ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
78762025-07-17T11:37:49.923ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78772025-07-17T11:37:49.925ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 143.929568ms iteration = 25 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
78782025-07-17T11:37:49.926ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78792025-07-17T11:37:49.926ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
78802025-07-17T11:37:49.926ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
78812025-07-17T11:37:49.927ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
78822025-07-17T11:37:49.929ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
78832025-07-17T11:37:49.930ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
78842025-07-17T11:37:49.930ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78852025-07-17T11:37:49.930ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
78862025-07-17T11:37:49.930ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
78872025-07-17T11:37:49.930ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
78882025-07-17T11:37:49.932ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78892025-07-17T11:37:49.932ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1194 zone_address = fd00:1122:3344:101::2
78902025-07-17T11:37:49.932ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
78912025-07-17T11:37:49.932ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78922025-07-17T11:37:49.933ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6ab55f8-a152-444e-92cc-f5f2dad9b757", "content-length": "26", "date": "Thu, 17 Jul 2025 11:37:49 GMT"} })
78932025-07-17T11:37:49.933ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1197 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
78942025-07-17T11:37:49.934ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
78952025-07-17T11:37:49.934ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
78962025-07-17T11:37:49.934ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
78972025-07-17T11:37:49.935ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
78982025-07-17T11:37:49.937ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
78992025-07-17T11:37:49.937ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
79002025-07-17T11:37:49.937ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
79012025-07-17T11:37:49.937ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
79022025-07-17T11:37:49.938ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
79032025-07-17T11:37:49.938ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:372
79042025-07-17T11:37:49.938ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 156.990079ms iteration = 25 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
79052025-07-17T11:37:49.938ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
79062025-07-17T11:37:49.939ZINFOnexus (BackgroundTasks): found Silo/DNS/TLS config (changed) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-07-17T11:25:46.964149Z, time_modified: 2025-07-17T11:25:46.964149Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 1fa836d9-7854-4ada-8c89-c26d922646c6, db_silo: Silo { identity: SiloIdentity { id: 1fa836d9-7854-4ada-8c89-c26d922646c6, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-07-17T11:26:35.643175Z, time_modified: 2025-07-17T11:26:35.643175Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "d7c3455b70a66c86ec2a7e597ca46fe0ea0c49dde5c1363f198c20aa77d58442" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 1fa836d9-7854-4ada-8c89-c26d922646c6, db_silo: Silo { identity: SiloIdentity { id: 1fa836d9-7854-4ada-8c89-c26d922646c6, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-07-17T11:26:35.643175Z, time_modified: 2025-07-17T11:26:35.643175Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "d7c3455b70a66c86ec2a7e597ca46fe0ea0c49dde5c1363f198c20aa77d58442" }] }) } file = nexus/src/app/background/tasks/external_endpoints.rs:94
79072025-07-17T11:37:49.939ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 261.563828ms iteration = 13 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"1fa836d9-7854-4ada-8c89-c26d922646c6","tls_certs":["d7c3455b70a66c86ec2a7e597ca46fe0ea0c49dde5c1363f198c20aa77d58442"]}},"default_endpoint":{"silo_id":"1fa836d9-7854-4ada-8c89-c26d922646c6","tls_certs":["d7c3455b70a66c86ec2a7e597ca46fe0ea0c49dde5c1363f198c20aa77d58442"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
79082025-07-17T11:37:49.941ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 4063, tv_nsec: 692540095 } })
79092025-07-17T11:37:49.942ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1186 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
79102025-07-17T11:37:49.951ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
79112025-07-17T11:37:49.951ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
79122025-07-17T11:37:49.967ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
79132025-07-17T11:37:49.967ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
79142025-07-17T11:37:49.967ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 186.449228ms iteration = 25 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
79152025-07-17T11:37:50.027ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
79162025-07-17T11:37:50.028ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
79172025-07-17T11:37:50.031ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
79182025-07-17T11:37:50.032ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
79192025-07-17T11:37:50.034ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79202025-07-17T11:37:50.040ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 0ae78f24-48db-4cf7-91cb-8d5311f2a17f, lookup_type: ById(0ae78f24-48db-4cf7-91cb-8d5311f2a17f) } result = Ok(())
79212025-07-17T11:37:50.040ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
79222025-07-17T11:37:50.040ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
79232025-07-17T11:37:50.043ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 262.4355ms iteration = 25 status = 0
79242025-07-17T11:37:50.110ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1194 zone_address = fd00:1122:3344:101::2
79252025-07-17T11:37:50.110ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
79262025-07-17T11:37:50.110ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "b509da08-8c20-47e1-a28c-33fb73b1b7e8", "content-length": "26", "date": "Thu, 17 Jul 2025 11:37:49 GMT"} })
79272025-07-17T11:37:50.110ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1197 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
79282025-07-17T11:37:50.112ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/all-sp-ids
79292025-07-17T11:37:50.112ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/all-sp-ids", status: 200, headers: {"content-type": "application/json", "x-request-id": "6946733f-9331-407a-970f-2c7e0941f361", "content-length": "78", "date": "Thu, 17 Jul 2025 11:37:49 GMT"} })
79302025-07-17T11:37:50.112ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = sled
79312025-07-17T11:37:50.112ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = switch
79322025-07-17T11:37:50.112ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 1 sp_type = sled
79332025-07-17T11:37:50.113ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
79342025-07-17T11:37:50.113ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = sled
79352025-07-17T11:37:50.113ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = switch
79362025-07-17T11:37:50.113ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 1 sp_type = sled
79372025-07-17T11:37:50.115ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 0 sp_type = sled
79382025-07-17T11:37:50.115ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 0 sp_type = switch
79392025-07-17T11:37:50.116ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 1 sp_type = sled
79402025-07-17T11:37:50.118ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 0 sp_type = sled
79412025-07-17T11:37:50.118ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = sled
79422025-07-17T11:37:50.118ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = sled
79432025-07-17T11:37:50.119ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 0 sp_type = switch
79442025-07-17T11:37:50.119ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = switch
79452025-07-17T11:37:50.119ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = switch
79462025-07-17T11:37:50.120ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 1 sp_type = sled
79472025-07-17T11:37:50.120ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 1 sp_type = sled
79482025-07-17T11:37:50.120ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 1 sp_type = sled
79492025-07-17T11:37:50.121ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 0 sp_type = sled start_ena = None
79502025-07-17T11:37:50.121ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
79512025-07-17T11:37:50.121ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "8fc1a572-3815-4c3c-ae0d-cab1e557ae4a", "content-length": "81", "date": "Thu, 17 Jul 2025 11:37:49 GMT"} })
79522025-07-17T11:37:50.121ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 0 sp_type = switch start_ena = None
79532025-07-17T11:37:50.121ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
79542025-07-17T11:37:50.122ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "735f2789-ddfb-4798-b357-7bc164a00c44", "content-length": "81", "date": "Thu, 17 Jul 2025 11:37:49 GMT"} })
79552025-07-17T11:37:50.136ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
79562025-07-17T11:37:50.137ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "c56c2499-0fec-41d3-9ed4-5bf258bbec2f", "content-length": "810", "date": "Thu, 17 Jul 2025 11:37:50 GMT"} })
79572025-07-17T11:37:50.137ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {VirtualNetworkInterfaceHost { physical_host_ip: fd00:1122:3344:101::1, virtual_ip: 172.30.0.5, virtual_mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), vni: Vni(3097446) }} sled = sock
79582025-07-17T11:37:50.137ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
79592025-07-17T11:37:50.137ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
79602025-07-17T11:37:50.144ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 362.787494ms iteration = 25 status = {}
79612025-07-17T11:37:50.161ZINFOnexus (BackgroundTasks): instance reincarnation completed successfully background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:113 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0
79622025-07-17T11:37:50.161ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 475.010813ms iteration = 13 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
79632025-07-17T11:37:50.164ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 383.346254ms iteration = 27 status = {"physical_disks_added":0}
79642025-07-17T11:37:50.174ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 1 sp_type = sled start_ena = None
79652025-07-17T11:37:50.174ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/1/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
79662025-07-17T11:37:50.175ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/1/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c182fa8-9ac6-4fc7-9f2e-09e543714ada", "content-length": "81", "date": "Thu, 17 Jul 2025 11:37:49 GMT"} })
79672025-07-17T11:37:50.177ZDEBGnexus (BackgroundTasks): activation complete background_task = sp_ereport_ingester elapsed = 395.831999ms iteration = 25 status = {"errors":[],"sps":[]}
79682025-07-17T11:37:50.182ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1194 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 zone_address = fd00:1122:3344:101::2
79692025-07-17T11:37:50.182ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
79702025-07-17T11:37:50.182ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "1aacd9d3-3cca-4530-bf26-0396be73913b", "content-length": "26", "date": "Thu, 17 Jul 2025 11:37:49 GMT"} })
79712025-07-17T11:37:50.182ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1197 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
79722025-07-17T11:37:50.184ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
79732025-07-17T11:37:50.229ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
79742025-07-17T11:37:50.229ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
79752025-07-17T11:37:50.232ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 553.6958ms iteration = 25 status = null
79762025-07-17T11:37:50.244ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = instance_watcher body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vmms/987a3b3d-06ca-492a-85db-b2d75a7b644b/state
79772025-07-17T11:37:50.246ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
79782025-07-17T11:37:50.308ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 573.670793ms iteration = 25 status = {}
79792025-07-17T11:37:50.313ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
79802025-07-17T11:37:50.313ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "6920666e-a329-4bd8-9914-01c3282509c6", "date": "Thu, 17 Jul 2025 11:37:50 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
79812025-07-17T11:37:50.313ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
79822025-07-17T11:37:50.315ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "d8f1c4ba-93b3-4227-ba3d-c034e44d124f", "date": "Thu, 17 Jul 2025 11:37:50 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
79832025-07-17T11:37:50.315ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
79842025-07-17T11:37:50.317ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
79852025-07-17T11:37:50.317ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 639.053055ms iteration = 15 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
79862025-07-17T11:37:50.373ZINFOnexus (BackgroundTasks): VPC route manager sled 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
79872025-07-17T11:37:50.373ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
79882025-07-17T11:37:50.373ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "6dfa05ee-7c7b-46f5-b7ff-ec547edf3d3b", "content-length": "523", "date": "Thu, 17 Jul 2025 11:37:50 GMT"} })
79892025-07-17T11:37:50.373ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
79902025-07-17T11:37:50.373ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
79912025-07-17T11:37:50.389ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {4ab3d5da-2db6-4d0d-88ff-d7618c67de1f: {10.151.2.176: {74fdf922-7d97-44ca-a0e8-778fa0911ce0}, 10.151.2.175: {74fdf922-7d97-44ca-a0e8-778fa0911ce0}}} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
79922025-07-17T11:37:50.389ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
79932025-07-17T11:37:50.390ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "25f5a713-143b-45ef-bb2b-455dbcca898b", "date": "Thu, 17 Jul 2025 11:37:50 GMT"} })
79942025-07-17T11:37:50.390ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
79952025-07-17T11:37:50.391ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
79962025-07-17T11:37:50.393ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
79972025-07-17T11:37:50.394ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "d8f5cd29-9994-4e5e-a147-c892c02f8701", "date": "Thu, 17 Jul 2025 11:37:50 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
79982025-07-17T11:37:50.394ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
79992025-07-17T11:37:50.394ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
80002025-07-17T11:37:50.394ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
80012025-07-17T11:37:50.395ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "4bc62152-03b0-4fdc-88ba-865379fa6f3a", "date": "Thu, 17 Jul 2025 11:37:50 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
80022025-07-17T11:37:50.395ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
80032025-07-17T11:37:50.397ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "92f2650e-c20b-4647-80b9-3585a616aff6", "date": "Thu, 17 Jul 2025 11:37:50 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
80042025-07-17T11:37:50.397ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
80052025-07-17T11:37:50.400ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
80062025-07-17T11:37:50.400ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 721.961779ms iteration = 15 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
80072025-07-17T11:37:50.402ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
80082025-07-17T11:37:50.402ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
80092025-07-17T11:37:50.406ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
80102025-07-17T11:37:50.407ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
80112025-07-17T11:37:50.409ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
80122025-07-17T11:37:50.409ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
80132025-07-17T11:37:50.412ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
80142025-07-17T11:37:50.412ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
80152025-07-17T11:37:50.415ZINFOnexus (BackgroundTasks): VPC route manager sled 0ae78f24-48db-4cf7-91cb-8d5311f2a17f push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
80162025-07-17T11:37:50.415ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
80172025-07-17T11:37:50.415ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
80182025-07-17T11:37:50.417ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
80192025-07-17T11:37:50.418ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
80202025-07-17T11:37:50.419ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
80212025-07-17T11:37:50.420ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
80222025-07-17T11:37:50.421ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
80232025-07-17T11:37:50.421ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
80242025-07-17T11:37:50.423ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
80252025-07-17T11:37:50.423ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
80262025-07-17T11:37:50.425ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
80272025-07-17T11:37:50.425ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
80282025-07-17T11:37:50.426ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
80292025-07-17T11:37:50.426ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
80302025-07-17T11:37:50.427ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80312025-07-17T11:37:50.431ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 1fa836d9-7854-4ada-8c89-c26d922646c6, lookup_type: ById(1fa836d9-7854-4ada-8c89-c26d922646c6) }, key: 4765c659-ba2a-4afe-8b90-7a2835ccc35f, lookup_type: ById(4765c659-ba2a-4afe-8b90-7a2835ccc35f) }, key: 053f1303-9c66-4e84-ab82-84adc9995d7f, lookup_type: ById(053f1303-9c66-4e84-ab82-84adc9995d7f) }, key: b17d433b-244d-4aa7-bc33-d35b6cf5cdcf, lookup_type: ById(b17d433b-244d-4aa7-bc33-d35b6cf5cdcf) } result = Ok(())
80322025-07-17T11:37:50.431ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
80332025-07-17T11:37:50.431ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
80342025-07-17T11:37:50.433ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
80352025-07-17T11:37:50.433ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
80362025-07-17T11:37:50.434ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
80372025-07-17T11:37:50.434ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
80382025-07-17T11:37:50.435ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80392025-07-17T11:37:50.439ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 1fa836d9-7854-4ada-8c89-c26d922646c6, lookup_type: ById(1fa836d9-7854-4ada-8c89-c26d922646c6) }, key: 4765c659-ba2a-4afe-8b90-7a2835ccc35f, lookup_type: ById(4765c659-ba2a-4afe-8b90-7a2835ccc35f) }, key: 053f1303-9c66-4e84-ab82-84adc9995d7f, lookup_type: ById(053f1303-9c66-4e84-ab82-84adc9995d7f) }, key: b17d433b-244d-4aa7-bc33-d35b6cf5cdcf, lookup_type: ById(b17d433b-244d-4aa7-bc33-d35b6cf5cdcf) } result = Ok(())
80402025-07-17T11:37:50.439ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
80412025-07-17T11:37:50.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
80422025-07-17T11:37:50.447ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
80432025-07-17T11:37:50.447ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
80442025-07-17T11:37:50.449ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 151caa7c-6a05-4995-94b4-7946d428117d, rack_id: 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5, switch_location: "switch0", port_name: Name(Name("qsfp0")), port_settings_id: Some(2e30c7ae-26da-46e4-9127-aac194921f20) } rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0
80452025-07-17T11:37:50.449ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
80462025-07-17T11:37:50.449ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
80472025-07-17T11:37:50.462ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1832 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = Name(Name("qsfp0"))
80482025-07-17T11:37:50.462ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1859 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0 switch_port_id = Name(Name("qsfp0")) switch_port_settings_id = 2e30c7ae-26da-46e4-9127-aac194921f20
80492025-07-17T11:37:50.462ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
80502025-07-17T11:37:50.462ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
80512025-07-17T11:37:50.462ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
80522025-07-17T11:37:50.462ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
80532025-07-17T11:37:50.462ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
80542025-07-17T11:37:50.462ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2006 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
80552025-07-17T11:37:50.462ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
80562025-07-17T11:37:50.462ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2042 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
80572025-07-17T11:37:50.462ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
80582025-07-17T11:37:50.462ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
80592025-07-17T11:37:50.463ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
80602025-07-17T11:37:50.464ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0
80612025-07-17T11:37:50.464ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
80622025-07-17T11:37:50.546ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
80632025-07-17T11:37:50.546ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
80642025-07-17T11:37:50.548ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
80652025-07-17T11:37:50.548ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
80662025-07-17T11:37:50.549ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
80672025-07-17T11:37:50.549ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
80682025-07-17T11:37:50.551ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
80692025-07-17T11:37:50.551ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
80702025-07-17T11:37:50.553ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
80712025-07-17T11:37:50.553ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
80722025-07-17T11:37:50.554ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
80732025-07-17T11:37:50.555ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
80742025-07-17T11:37:50.556ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
80752025-07-17T11:37:50.556ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
80762025-07-17T11:37:50.558ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80772025-07-17T11:37:50.561ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 1fa836d9-7854-4ada-8c89-c26d922646c6, lookup_type: ById(1fa836d9-7854-4ada-8c89-c26d922646c6) }, key: 4765c659-ba2a-4afe-8b90-7a2835ccc35f, lookup_type: ById(4765c659-ba2a-4afe-8b90-7a2835ccc35f) }, key: 053f1303-9c66-4e84-ab82-84adc9995d7f, lookup_type: ById(053f1303-9c66-4e84-ab82-84adc9995d7f) }, key: 3c0de112-a4af-4029-a7bc-1f5ed68a73a1, lookup_type: ById(3c0de112-a4af-4029-a7bc-1f5ed68a73a1) } result = Ok(())
80782025-07-17T11:37:50.564ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
80792025-07-17T11:37:50.564ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
80802025-07-17T11:37:50.567ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
80812025-07-17T11:37:50.567ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
80822025-07-17T11:37:50.569ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
80832025-07-17T11:37:50.569ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
80842025-07-17T11:37:50.570ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
80852025-07-17T11:37:50.570ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
80862025-07-17T11:37:50.572ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
80872025-07-17T11:37:50.572ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
80882025-07-17T11:37:50.574ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
80892025-07-17T11:37:50.574ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
80902025-07-17T11:37:50.575ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
80912025-07-17T11:37:50.575ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
80922025-07-17T11:37:50.577ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80932025-07-17T11:37:50.579ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "be7387f6-bd42-4b22-8cfe-a9751b2ceea3", "date": "Thu, 17 Jul 2025 11:37:50 GMT"} })
80942025-07-17T11:37:50.579ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch = Switch0
80952025-07-17T11:37:50.580ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
80962025-07-17T11:37:50.580ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 1fa836d9-7854-4ada-8c89-c26d922646c6, lookup_type: ById(1fa836d9-7854-4ada-8c89-c26d922646c6) }, key: 4765c659-ba2a-4afe-8b90-7a2835ccc35f, lookup_type: ById(4765c659-ba2a-4afe-8b90-7a2835ccc35f) }, key: 053f1303-9c66-4e84-ab82-84adc9995d7f, lookup_type: ById(053f1303-9c66-4e84-ab82-84adc9995d7f) }, key: 3c0de112-a4af-4029-a7bc-1f5ed68a73a1, lookup_type: ByName("default") } result = Ok(())
80972025-07-17T11:37:50.580ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
80982025-07-17T11:37:50.580ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
80992025-07-17T11:37:50.582ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
81002025-07-17T11:37:50.582ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
81012025-07-17T11:37:50.584ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
81022025-07-17T11:37:50.584ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
81032025-07-17T11:37:50.585ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
81042025-07-17T11:37:50.585ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
81052025-07-17T11:37:50.586ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "df86cb02-0cc0-48e3-815b-cd579bb289bf", "content-length": "682", "date": "Thu, 17 Jul 2025 11:37:50 GMT"} })
81062025-07-17T11:37:50.586ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
81072025-07-17T11:37:50.586ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
81082025-07-17T11:37:50.589ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
81092025-07-17T11:37:50.589ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
81102025-07-17T11:37:50.592ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
81112025-07-17T11:37:50.592ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
81122025-07-17T11:37:50.592ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
81132025-07-17T11:37:50.592ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
81142025-07-17T11:37:50.593ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
81152025-07-17T11:37:50.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
81162025-07-17T11:37:50.593ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
81172025-07-17T11:37:50.593ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
81182025-07-17T11:37:50.595ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
81192025-07-17T11:37:50.595ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
81202025-07-17T11:37:50.596ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81212025-07-17T11:37:50.600ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 818.848735ms iteration = 25 status = {}
81222025-07-17T11:37:50.602ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = InternetGateway { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 1fa836d9-7854-4ada-8c89-c26d922646c6, lookup_type: ById(1fa836d9-7854-4ada-8c89-c26d922646c6) }, key: 4765c659-ba2a-4afe-8b90-7a2835ccc35f, lookup_type: ById(4765c659-ba2a-4afe-8b90-7a2835ccc35f) }, key: 053f1303-9c66-4e84-ab82-84adc9995d7f, lookup_type: ById(053f1303-9c66-4e84-ab82-84adc9995d7f) }, key: 74fdf922-7d97-44ca-a0e8-778fa0911ce0, lookup_type: ByName("default") } result = Ok(())
81232025-07-17T11:37:50.613ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
81242025-07-17T11:37:50.615ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "d90ba123-36a8-4ad8-9274-14b49c1c9dfe", "date": "Thu, 17 Jul 2025 11:37:50 GMT"} })
81252025-07-17T11:37:50.616ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 835.240792ms iteration = 25 status = {}
81262025-07-17T11:37:51.694ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 321 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = 4788f26c-fbd7-411f-be7d-c43df03c48b5 response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
81272025-07-17T11:37:53.695ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = instance_watcher result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vmms/987a3b3d-06ca-492a-85db-b2d75a7b644b/state", status: 200, headers: {"content-type": "application/json", "x-request-id": "32f109c5-39ea-4b46-b68a-c01a15c5c377", "content-length": "131", "date": "Thu, 17 Jul 2025 11:37:53 GMT"} })
81282025-07-17T11:37:53.695ZDEBGnexus (BackgroundTasks): updating instance state background_task = instance_watcher instance_id = 0a6645df-a4d8-457d-81a4-db94f3aeb746 sled_id = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f state = SledVmmState { vmm_state: VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2025-07-17T11:37:45.808488708Z }, migration_in: None, migration_out: None } vmm_id = 987a3b3d-06ca-492a-85db-b2d75a7b644b
81292025-07-17T11:37:53.695ZINFOnexus (BackgroundTasks): received new VMM runtime state from sled agent background_task = instance_watcher file = nexus/src/app/instance.rs:2140 instance_id = 0a6645df-a4d8-457d-81a4-db94f3aeb746 migration_state = Migrations { migration_in: None, migration_out: None } propolis_id = 987a3b3d-06ca-492a-85db-b2d75a7b644b sled_id = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f vmm_id = 987a3b3d-06ca-492a-85db-b2d75a7b644b vmm_state = VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2025-07-17T11:37:45.808488708Z }
81302025-07-17T11:37:53.695ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher instance_id = 0a6645df-a4d8-457d-81a4-db94f3aeb746 roles = RoleSet { roles: {} } sled_id = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f vmm_id = 987a3b3d-06ca-492a-85db-b2d75a7b644b
81312025-07-17T11:37:53.695ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher instance_id = 0a6645df-a4d8-457d-81a4-db94f3aeb746 resource = Database result = Ok(()) sled_id = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f vmm_id = 987a3b3d-06ca-492a-85db-b2d75a7b644b
81322025-07-17T11:37:53.708ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:550 pruned_instances = 0 total_completed = 1 total_failed = 0 total_incomplete = 0 total_instances = 1 update_sagas_queued = 0
81332025-07-17T11:37:53.710ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 3.929052649s iteration = 25 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{"starting":1},"pruned_instances":0,"total_instances":1,"update_sagas_queued":0}
81342025-07-17T11:37:54.782ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 146 reason = Timeout
81352025-07-17T11:37:54.782ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
81362025-07-17T11:37:54.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
81372025-07-17T11:37:54.784ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81382025-07-17T11:37:54.787ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
81392025-07-17T11:37:54.787ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
81402025-07-17T11:37:54.787ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
81412025-07-17T11:37:54.789ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
81422025-07-17T11:37:54.789ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 7.331146ms iteration = 146 status = {"chicken_switches_updated":false}
81432025-07-17T11:37:59.782ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 74 reason = Timeout
81442025-07-17T11:37:59.782ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 74 reason = Timeout
81452025-07-17T11:37:59.782ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
81462025-07-17T11:37:59.782ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
81472025-07-17T11:37:59.782ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 147 reason = Timeout
81482025-07-17T11:37:59.782ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
81492025-07-17T11:37:59.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
81502025-07-17T11:37:59.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
81512025-07-17T11:37:59.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
81522025-07-17T11:37:59.784ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81532025-07-17T11:37:59.784ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81542025-07-17T11:37:59.785ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
81552025-07-17T11:37:59.785ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
81562025-07-17T11:37:59.787ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 5.857787ms iteration = 74 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
81572025-07-17T11:37:59.788ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
81582025-07-17T11:37:59.788ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
81592025-07-17T11:37:59.788ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
81602025-07-17T11:37:59.788ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
81612025-07-17T11:37:59.788ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
81622025-07-17T11:37:59.788ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
81632025-07-17T11:37:59.790ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
81642025-07-17T11:37:59.790ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
81652025-07-17T11:37:59.791ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
81662025-07-17T11:37:59.791ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 9.036229ms iteration = 147 status = {"chicken_switches_updated":false}
81672025-07-17T11:37:59.791ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81682025-07-17T11:37:59.794ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
81692025-07-17T11:37:59.794ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
81702025-07-17T11:37:59.794ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
81712025-07-17T11:37:59.807ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
81722025-07-17T11:37:59.807ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 25.04701ms iteration = 74 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
81732025-07-17T11:38:01.685ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::d]:64540
81742025-07-17T11:38:01.685ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:64540 req_id = 9ddee79e-3ff3-4fc9-8153-99c55ebcbc91 roles = RoleSet { roles: {} } uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?sort_by=id_ascending
81752025-07-17T11:38:01.686ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:64540 req_id = 9ddee79e-3ff3-4fc9-8153-99c55ebcbc91 resource = Database result = Ok(()) uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?sort_by=id_ascending
81762025-07-17T11:38:01.688ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 2175 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:64540 req_id = 9ddee79e-3ff3-4fc9-8153-99c55ebcbc91 response_code = 200 uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?sort_by=id_ascending
81772025-07-17T11:38:01.688ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:64540 req_id = 1dd8d94a-4b2a-440f-86cc-0af55b82e6b8 roles = RoleSet { roles: {} } uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmMzlkM2U3MC02NmU0LTQ2OGEtYTQ0ZC03NDZkMjJiZWEzYTAifX0%3D
81782025-07-17T11:38:01.688ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:64540 req_id = 1dd8d94a-4b2a-440f-86cc-0af55b82e6b8 resource = Database result = Ok(()) uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmMzlkM2U3MC02NmU0LTQ2OGEtYTQ0ZC03NDZkMjJiZWEzYTAifX0%3D
81792025-07-17T11:38:01.690ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 2302 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:64540 req_id = 1dd8d94a-4b2a-440f-86cc-0af55b82e6b8 response_code = 200 uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmMzlkM2U3MC02NmU0LTQ2OGEtYTQ0ZC03NDZkMjJiZWEzYTAifX0%3D
81802025-07-17T11:38:01.694ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 373 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = 781cd6db-47d5-4a09-bf3e-570916814066 response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
81812025-07-17T11:38:04.782ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 148 reason = Timeout
81822025-07-17T11:38:04.782ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
81832025-07-17T11:38:04.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
81842025-07-17T11:38:04.784ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81852025-07-17T11:38:04.787ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
81862025-07-17T11:38:04.787ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
81872025-07-17T11:38:04.787ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
81882025-07-17T11:38:04.789ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
81892025-07-17T11:38:04.789ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 6.908731ms iteration = 148 status = {"chicken_switches_updated":false}
81902025-07-17T11:38:09.782ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 75 reason = Timeout
81912025-07-17T11:38:09.782ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 75 reason = Timeout
81922025-07-17T11:38:09.782ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
81932025-07-17T11:38:09.782ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 149 reason = Timeout
81942025-07-17T11:38:09.782ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
81952025-07-17T11:38:09.782ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
81962025-07-17T11:38:09.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
81972025-07-17T11:38:09.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
81982025-07-17T11:38:09.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
81992025-07-17T11:38:09.784ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
82002025-07-17T11:38:09.785ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
82012025-07-17T11:38:09.785ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82022025-07-17T11:38:09.786ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82032025-07-17T11:38:09.787ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.611881ms iteration = 75 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
82042025-07-17T11:38:09.788ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
82052025-07-17T11:38:09.788ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
82062025-07-17T11:38:09.788ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
82072025-07-17T11:38:09.788ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
82082025-07-17T11:38:09.788ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
82092025-07-17T11:38:09.788ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
82102025-07-17T11:38:09.790ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
82112025-07-17T11:38:09.790ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
82122025-07-17T11:38:09.790ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 8.249809ms iteration = 149 status = {"chicken_switches_updated":false}
82132025-07-17T11:38:09.790ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
82142025-07-17T11:38:09.792ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82152025-07-17T11:38:09.795ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
82162025-07-17T11:38:09.795ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
82172025-07-17T11:38:09.795ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
82182025-07-17T11:38:09.804ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
82192025-07-17T11:38:09.804ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 22.585648ms iteration = 75 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
82202025-07-17T11:38:11.694ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 636 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = 3f3ba995-61a9-407f-9b9e-27f443c9a08a response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
82212025-07-17T11:38:14.782ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 150 reason = Timeout
82222025-07-17T11:38:14.782ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
82232025-07-17T11:38:14.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
82242025-07-17T11:38:14.784ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82252025-07-17T11:38:14.787ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
82262025-07-17T11:38:14.787ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
82272025-07-17T11:38:14.787ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
82282025-07-17T11:38:14.789ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
82292025-07-17T11:38:14.789ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 6.788359ms iteration = 150 status = {"chicken_switches_updated":false}
82302025-07-17T11:38:19.677ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 26 reason = Timeout
82312025-07-17T11:38:19.677ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
82322025-07-17T11:38:19.678ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
82332025-07-17T11:38:19.680ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 4093, tv_nsec: 430817839 } })
82342025-07-17T11:38:19.680ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1186
82352025-07-17T11:38:19.710ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1194 zone_address = fd00:1122:3344:101::2
82362025-07-17T11:38:19.710ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
82372025-07-17T11:38:19.710ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "93e0a511-848b-40cc-ab80-dac84350beeb", "content-length": "26", "date": "Thu, 17 Jul 2025 11:38:19 GMT"} })
82382025-07-17T11:38:19.710ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1197 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
82392025-07-17T11:38:19.711ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
82402025-07-17T11:38:19.735ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 26 reason = Timeout
82412025-07-17T11:38:19.735ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 26 reason = Timeout
82422025-07-17T11:38:19.735ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
82432025-07-17T11:38:19.735ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
82442025-07-17T11:38:19.737ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 2.383371ms iteration = 26 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
82452025-07-17T11:38:19.738ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 4093, tv_nsec: 488867465 } })
82462025-07-17T11:38:19.738ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1186
82472025-07-17T11:38:19.741ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
82482025-07-17T11:38:19.741ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
82492025-07-17T11:38:19.742ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 65.041067ms iteration = 26 status = null
82502025-07-17T11:38:19.770ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1194 zone_address = fd00:1122:3344:101::2
82512025-07-17T11:38:19.770ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
82522025-07-17T11:38:19.771ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "98b242e9-7a5a-48b8-b308-62f2be834627", "content-length": "26", "date": "Thu, 17 Jul 2025 11:38:19 GMT"} })
82532025-07-17T11:38:19.771ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1197 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
82542025-07-17T11:38:19.772ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
82552025-07-17T11:38:19.782ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 26 reason = Timeout
82562025-07-17T11:38:19.782ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 26 reason = Timeout
82572025-07-17T11:38:19.782ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
82582025-07-17T11:38:19.782ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
82592025-07-17T11:38:19.782ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 26 reason = Timeout
82602025-07-17T11:38:19.782ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
82612025-07-17T11:38:19.782ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 26 reason = Timeout
82622025-07-17T11:38:19.782ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 26 reason = Timeout
82632025-07-17T11:38:19.782ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
82642025-07-17T11:38:19.782ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
82652025-07-17T11:38:19.782ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 26 reason = Timeout
82662025-07-17T11:38:19.782ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
82672025-07-17T11:38:19.782ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 26 reason = Timeout
82682025-07-17T11:38:19.782ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
82692025-07-17T11:38:19.782ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
82702025-07-17T11:38:19.782ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 26 reason = Timeout
82712025-07-17T11:38:19.782ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
82722025-07-17T11:38:19.782ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 76 reason = Timeout
82732025-07-17T11:38:19.782ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
82742025-07-17T11:38:19.782ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
82752025-07-17T11:38:19.782ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 26 reason = Timeout
82762025-07-17T11:38:19.782ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
82772025-07-17T11:38:19.782ZDEBGnexus (BackgroundTasks): activating background_task = sp_ereport_ingester iteration = 26 reason = Timeout
82782025-07-17T11:38:19.782ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 26 reason = Timeout
82792025-07-17T11:38:19.782ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
82802025-07-17T11:38:19.782ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 28 reason = Timeout
82812025-07-17T11:38:19.782ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
82822025-07-17T11:38:19.782ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 76 reason = Timeout
82832025-07-17T11:38:19.782ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 151 reason = Timeout
82842025-07-17T11:38:19.782ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
82852025-07-17T11:38:19.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
82862025-07-17T11:38:19.782ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
82872025-07-17T11:38:19.782ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 26 reason = Timeout
82882025-07-17T11:38:19.782ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 26 reason = Timeout
82892025-07-17T11:38:19.782ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
82902025-07-17T11:38:19.782ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
82912025-07-17T11:38:19.782ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 26 reason = Timeout
82922025-07-17T11:38:19.782ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
82932025-07-17T11:38:19.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
82942025-07-17T11:38:19.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
82952025-07-17T11:38:19.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
82962025-07-17T11:38:19.783ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
82972025-07-17T11:38:19.783ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12225, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 4093, tv_nsec: 533699666 } })
82982025-07-17T11:38:19.783ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
82992025-07-17T11:38:19.783ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
83002025-07-17T11:38:19.783ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
83012025-07-17T11:38:19.783ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
83022025-07-17T11:38:19.783ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
83032025-07-17T11:38:19.783ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
83042025-07-17T11:38:19.783ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
83052025-07-17T11:38:19.783ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
83062025-07-17T11:38:19.783ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
83072025-07-17T11:38:19.783ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
83082025-07-17T11:38:19.783ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
83092025-07-17T11:38:19.783ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
83102025-07-17T11:38:19.785ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 2.860358ms iteration = 26 status = {"errors":[],"garbage_collect_requested":[]}
83112025-07-17T11:38:19.787ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
83122025-07-17T11:38:19.787ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83132025-07-17T11:38:19.787ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
83142025-07-17T11:38:19.788ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83152025-07-17T11:38:19.788ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83162025-07-17T11:38:19.789ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83172025-07-17T11:38:19.789ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83182025-07-17T11:38:19.790ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83192025-07-17T11:38:19.790ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83202025-07-17T11:38:19.790ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
83212025-07-17T11:38:19.790ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
83222025-07-17T11:38:19.791ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
83232025-07-17T11:38:19.791ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83242025-07-17T11:38:19.791ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
83252025-07-17T11:38:19.791ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
83262025-07-17T11:38:19.791ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
83272025-07-17T11:38:19.792ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
83282025-07-17T11:38:19.792ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
83292025-07-17T11:38:19.793ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
83302025-07-17T11:38:19.793ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
83312025-07-17T11:38:19.793ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
83322025-07-17T11:38:19.793ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 4093, tv_nsec: 544036412 } })
83332025-07-17T11:38:19.793ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
83342025-07-17T11:38:19.793ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
83352025-07-17T11:38:19.793ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
83362025-07-17T11:38:19.793ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1186 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
83372025-07-17T11:38:19.794ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
83382025-07-17T11:38:19.794ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
83392025-07-17T11:38:19.794ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
83402025-07-17T11:38:19.796ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
83412025-07-17T11:38:19.796ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
83422025-07-17T11:38:19.796ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
83432025-07-17T11:38:19.796ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
83442025-07-17T11:38:19.796ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
83452025-07-17T11:38:19.796ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
83462025-07-17T11:38:19.796ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
83472025-07-17T11:38:19.796ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
83482025-07-17T11:38:19.796ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
83492025-07-17T11:38:19.797ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
83502025-07-17T11:38:19.797ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
83512025-07-17T11:38:19.797ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
83522025-07-17T11:38:19.797ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
83532025-07-17T11:38:19.798ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
83542025-07-17T11:38:19.799ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83552025-07-17T11:38:19.799ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
83562025-07-17T11:38:19.800ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
83572025-07-17T11:38:19.801ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
83582025-07-17T11:38:19.802ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
83592025-07-17T11:38:19.802ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
83602025-07-17T11:38:19.802ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
83612025-07-17T11:38:19.802ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 19.295904ms iteration = 151 status = {"chicken_switches_updated":false}
83622025-07-17T11:38:19.802ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
83632025-07-17T11:38:19.802ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
83642025-07-17T11:38:19.802ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
83652025-07-17T11:38:19.802ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
83662025-07-17T11:38:19.803ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
83672025-07-17T11:38:19.803ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83682025-07-17T11:38:19.805ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
83692025-07-17T11:38:19.805ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
83702025-07-17T11:38:19.806ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
83712025-07-17T11:38:19.806ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
83722025-07-17T11:38:19.807ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
83732025-07-17T11:38:19.807ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83742025-07-17T11:38:19.812ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
83752025-07-17T11:38:19.812ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
83762025-07-17T11:38:19.812ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
83772025-07-17T11:38:19.829ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
83782025-07-17T11:38:19.829ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 47.142141ms iteration = 76 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
83792025-07-17T11:38:19.861ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 78.962921ms iteration = 26 status = {"errors":[],"finish_invoked_ok":[]}
83802025-07-17T11:38:19.878ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 96.33221ms iteration = 76 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
83812025-07-17T11:38:19.879ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
83822025-07-17T11:38:19.879ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 144.031339ms iteration = 26 status = {}
83832025-07-17T11:38:19.879ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 96.767165ms iteration = 26 status = {"errors":[],"requests_created_ok":[]}
83842025-07-17T11:38:19.879ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
83852025-07-17T11:38:19.887ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
83862025-07-17T11:38:19.887ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
83872025-07-17T11:38:19.889ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
83882025-07-17T11:38:19.890ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
83892025-07-17T11:38:19.890ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
83902025-07-17T11:38:19.890ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
83912025-07-17T11:38:19.892ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 109.612845ms iteration = 26 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
83922025-07-17T11:38:19.897ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
83932025-07-17T11:38:19.897ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
83942025-07-17T11:38:19.897ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 114.809674ms iteration = 26 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
83952025-07-17T11:38:19.899ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83962025-07-17T11:38:19.900ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
83972025-07-17T11:38:19.900ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
83982025-07-17T11:38:19.900ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 117.683251ms iteration = 26 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
83992025-07-17T11:38:19.902ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
84002025-07-17T11:38:19.903ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
84012025-07-17T11:38:19.903ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
84022025-07-17T11:38:19.903ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
84032025-07-17T11:38:19.904ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
84042025-07-17T11:38:19.906ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
84052025-07-17T11:38:19.906ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:550 pruned_instances = 1 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
84062025-07-17T11:38:19.906ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 123.951735ms iteration = 26 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":1,"total_instances":0,"update_sagas_queued":0}
84072025-07-17T11:38:19.906ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
84082025-07-17T11:38:19.908ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84092025-07-17T11:38:19.912ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 0ae78f24-48db-4cf7-91cb-8d5311f2a17f, lookup_type: ById(0ae78f24-48db-4cf7-91cb-8d5311f2a17f) } result = Ok(())
84102025-07-17T11:38:19.912ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
84112025-07-17T11:38:19.913ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
84122025-07-17T11:38:19.915ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 132.583258ms iteration = 26 status = 0
84132025-07-17T11:38:19.918ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/all-sp-ids
84142025-07-17T11:38:19.919ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/all-sp-ids", status: 200, headers: {"content-type": "application/json", "x-request-id": "5148e66c-ad6e-4fe1-bd48-05b6f8bfac86", "content-length": "78", "date": "Thu, 17 Jul 2025 11:38:19 GMT"} })
84152025-07-17T11:38:19.919ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = switch
84162025-07-17T11:38:19.919ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 1 sp_type = sled
84172025-07-17T11:38:19.919ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = sled
84182025-07-17T11:38:19.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 1 sp_type = sled
84192025-07-17T11:38:19.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = sled
84202025-07-17T11:38:19.920ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = switch
84212025-07-17T11:38:19.925ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 1 sp_type = sled
84222025-07-17T11:38:19.925ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 0 sp_type = sled
84232025-07-17T11:38:19.926ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 0 sp_type = switch
84242025-07-17T11:38:19.929ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 1 sp_type = sled
84252025-07-17T11:38:19.929ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 1 sp_type = sled
84262025-07-17T11:38:19.930ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 1 sp_type = sled
84272025-07-17T11:38:19.930ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 0 sp_type = switch
84282025-07-17T11:38:19.930ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = switch
84292025-07-17T11:38:19.930ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 0 sp_type = sled
84302025-07-17T11:38:19.930ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = sled
84312025-07-17T11:38:19.930ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = switch
84322025-07-17T11:38:19.930ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = sled
84332025-07-17T11:38:19.933ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 1 sp_type = sled start_ena = None
84342025-07-17T11:38:19.933ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/1/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
84352025-07-17T11:38:19.934ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/1/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ee5b6ec-1573-4f85-b7b5-fa7b78b72aca", "content-length": "81", "date": "Thu, 17 Jul 2025 11:38:19 GMT"} })
84362025-07-17T11:38:19.934ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1194 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 zone_address = fd00:1122:3344:101::2
84372025-07-17T11:38:19.934ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
84382025-07-17T11:38:19.934ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "fae5d801-b956-46ad-b64f-f53177ca4786", "content-length": "26", "date": "Thu, 17 Jul 2025 11:38:19 GMT"} })
84392025-07-17T11:38:19.934ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1197 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
84402025-07-17T11:38:19.935ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 0 sp_type = switch start_ena = None
84412025-07-17T11:38:19.935ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
84422025-07-17T11:38:19.935ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "db23d725-3fce-49dd-9ec4-baa17466eab1", "content-length": "81", "date": "Thu, 17 Jul 2025 11:38:19 GMT"} })
84432025-07-17T11:38:19.936ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 0 sp_type = sled start_ena = None
84442025-07-17T11:38:19.936ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
84452025-07-17T11:38:19.936ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "0416ac64-aadc-4cc7-8a99-28b98e4c6149", "content-length": "81", "date": "Thu, 17 Jul 2025 11:38:19 GMT"} })
84462025-07-17T11:38:19.936ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
84472025-07-17T11:38:19.938ZDEBGnexus (BackgroundTasks): activation complete background_task = sp_ereport_ingester elapsed = 155.972556ms iteration = 26 status = {"errors":[],"sps":[]}
84482025-07-17T11:38:19.952ZINFOnexus (BackgroundTasks): VPC route manager sled 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
84492025-07-17T11:38:19.952ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
84502025-07-17T11:38:19.953ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b5f59fb-146c-4f21-9b55-1e8b06e33b85", "content-length": "372", "date": "Thu, 17 Jul 2025 11:38:19 GMT"} })
84512025-07-17T11:38:19.953ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
84522025-07-17T11:38:19.953ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
84532025-07-17T11:38:19.964ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 181.552784ms iteration = 26 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
84542025-07-17T11:38:19.981ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:372
84552025-07-17T11:38:19.981ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 199.260247ms iteration = 26 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
84562025-07-17T11:38:20.004ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
84572025-07-17T11:38:20.004ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
84582025-07-17T11:38:20.006ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84592025-07-17T11:38:20.008ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
84602025-07-17T11:38:20.008ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
84612025-07-17T11:38:20.008ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
84622025-07-17T11:38:20.010ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84632025-07-17T11:38:20.012ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
84642025-07-17T11:38:20.012ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
84652025-07-17T11:38:20.012ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
84662025-07-17T11:38:20.017ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
84672025-07-17T11:38:20.017ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
84682025-07-17T11:38:20.017ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "1ec76649-e73d-4637-bf35-05ac4e1a91aa", "date": "Thu, 17 Jul 2025 11:38:19 GMT"} })
84692025-07-17T11:38:20.017ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
84702025-07-17T11:38:20.018ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
84712025-07-17T11:38:20.020ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
84722025-07-17T11:38:20.020ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
84732025-07-17T11:38:20.023ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 240.697274ms iteration = 28 status = {"physical_disks_added":0}
84742025-07-17T11:38:20.023ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
84752025-07-17T11:38:20.023ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
84762025-07-17T11:38:20.026ZINFOnexus (BackgroundTasks): VPC route manager sled 0ae78f24-48db-4cf7-91cb-8d5311f2a17f push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
84772025-07-17T11:38:20.026ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
84782025-07-17T11:38:20.026ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "5abbbd0a-5c00-4c8a-886f-abb6ed348d25", "date": "Thu, 17 Jul 2025 11:38:20 GMT"} })
84792025-07-17T11:38:20.028ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 246.332167ms iteration = 26 status = {}
84802025-07-17T11:38:20.074ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
84812025-07-17T11:38:20.075ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "bdad63cd-6345-43d5-ac2f-b5401d55c905", "content-length": "691", "date": "Thu, 17 Jul 2025 11:38:20 GMT"} })
84822025-07-17T11:38:20.075ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
84832025-07-17T11:38:20.075ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
84842025-07-17T11:38:20.075ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
84852025-07-17T11:38:20.077ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 295.049549ms iteration = 26 status = {}
84862025-07-17T11:38:20.093ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
84872025-07-17T11:38:20.094ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
84882025-07-17T11:38:20.096ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 151caa7c-6a05-4995-94b4-7946d428117d, rack_id: 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5, switch_location: "switch0", port_name: Name(Name("qsfp0")), port_settings_id: Some(2e30c7ae-26da-46e4-9127-aac194921f20) } rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0
84892025-07-17T11:38:20.096ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
84902025-07-17T11:38:20.096ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
84912025-07-17T11:38:20.107ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1832 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = Name(Name("qsfp0"))
84922025-07-17T11:38:20.107ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1859 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0 switch_port_id = Name(Name("qsfp0")) switch_port_settings_id = 2e30c7ae-26da-46e4-9127-aac194921f20
84932025-07-17T11:38:20.107ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
84942025-07-17T11:38:20.107ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
84952025-07-17T11:38:20.107ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
84962025-07-17T11:38:20.107ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
84972025-07-17T11:38:20.107ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
84982025-07-17T11:38:20.107ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2006 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
84992025-07-17T11:38:20.107ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
85002025-07-17T11:38:20.107ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2042 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
85012025-07-17T11:38:20.107ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
85022025-07-17T11:38:20.107ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
85032025-07-17T11:38:20.108ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
85042025-07-17T11:38:20.110ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0
85052025-07-17T11:38:20.110ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
85062025-07-17T11:38:20.285ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "649a1614-81d5-48d8-a63d-ae9d4ab1666c", "date": "Thu, 17 Jul 2025 11:38:20 GMT"} })
85072025-07-17T11:38:20.285ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch = Switch0
85082025-07-17T11:38:20.285ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
85092025-07-17T11:38:20.292ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "48ab07cd-7fd7-4b41-9cdd-0c13c61a9b20", "content-length": "682", "date": "Thu, 17 Jul 2025 11:38:20 GMT"} })
85102025-07-17T11:38:20.292ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
85112025-07-17T11:38:20.293ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
85122025-07-17T11:38:20.296ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
85132025-07-17T11:38:20.296ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
85142025-07-17T11:38:20.299ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
85152025-07-17T11:38:20.299ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
85162025-07-17T11:38:20.301ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
85172025-07-17T11:38:20.301ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
85182025-07-17T11:38:20.308ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 525.781612ms iteration = 26 status = {}
85192025-07-17T11:38:21.694ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 383 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = d3299a1f-8e0f-41c2-bba0-2f22248cf874 response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
85202025-07-17T11:38:24.782ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 152 reason = Timeout
85212025-07-17T11:38:24.782ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
85222025-07-17T11:38:24.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
85232025-07-17T11:38:24.784ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85242025-07-17T11:38:24.787ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
85252025-07-17T11:38:24.787ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
85262025-07-17T11:38:24.787ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
85272025-07-17T11:38:24.789ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
85282025-07-17T11:38:24.789ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 7.324376ms iteration = 152 status = {"chicken_switches_updated":false}
85292025-07-17T11:38:29.781ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 77 reason = Timeout
85302025-07-17T11:38:29.782ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 77 reason = Timeout
85312025-07-17T11:38:29.782ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
85322025-07-17T11:38:29.782ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
85332025-07-17T11:38:29.782ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 153 reason = Timeout
85342025-07-17T11:38:29.782ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
85352025-07-17T11:38:29.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
85362025-07-17T11:38:29.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
85372025-07-17T11:38:29.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
85382025-07-17T11:38:29.784ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85392025-07-17T11:38:29.785ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85402025-07-17T11:38:29.786ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
85412025-07-17T11:38:29.786ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
85422025-07-17T11:38:29.787ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
85432025-07-17T11:38:29.787ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
85442025-07-17T11:38:29.787ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
85452025-07-17T11:38:29.788ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 6.465746ms iteration = 77 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
85462025-07-17T11:38:29.788ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
85472025-07-17T11:38:29.788ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
85482025-07-17T11:38:29.788ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
85492025-07-17T11:38:29.789ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
85502025-07-17T11:38:29.789ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
85512025-07-17T11:38:29.789ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 7.874864ms iteration = 153 status = {"chicken_switches_updated":false}
85522025-07-17T11:38:29.789ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
85532025-07-17T11:38:29.791ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85542025-07-17T11:38:29.793ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
85552025-07-17T11:38:29.793ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
85562025-07-17T11:38:29.794ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
85572025-07-17T11:38:29.807ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
85582025-07-17T11:38:29.807ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 25.720299ms iteration = 77 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
85592025-07-17T11:38:31.686ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:64540 req_id = a6e23305-cc7e-4d36-a380-341bcd87b400 roles = RoleSet { roles: {} } uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?sort_by=id_ascending
85602025-07-17T11:38:31.686ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:64540 req_id = a6e23305-cc7e-4d36-a380-341bcd87b400 resource = Database result = Ok(()) uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?sort_by=id_ascending
85612025-07-17T11:38:31.689ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 3618 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:64540 req_id = a6e23305-cc7e-4d36-a380-341bcd87b400 response_code = 200 uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?sort_by=id_ascending
85622025-07-17T11:38:31.689ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:64540 req_id = 22585b8d-2313-4e58-901b-135eb4d65b3e roles = RoleSet { roles: {} } uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmMzlkM2U3MC02NmU0LTQ2OGEtYTQ0ZC03NDZkMjJiZWEzYTAifX0%3D
85632025-07-17T11:38:31.690ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:64540 req_id = 22585b8d-2313-4e58-901b-135eb4d65b3e resource = Database result = Ok(()) uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmMzlkM2U3MC02NmU0LTQ2OGEtYTQ0ZC03NDZkMjJiZWEzYTAifX0%3D
85642025-07-17T11:38:31.692ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 2496 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:64540 req_id = 22585b8d-2313-4e58-901b-135eb4d65b3e response_code = 200 uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmMzlkM2U3MC02NmU0LTQ2OGEtYTQ0ZC03NDZkMjJiZWEzYTAifX0%3D
85652025-07-17T11:38:31.693ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 321 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = a84998ef-b315-420a-87c6-37f64a0ae4c8 response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
85662025-07-17T11:38:34.782ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 154 reason = Timeout
85672025-07-17T11:38:34.782ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
85682025-07-17T11:38:34.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
85692025-07-17T11:38:34.783ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85702025-07-17T11:38:34.786ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
85712025-07-17T11:38:34.786ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
85722025-07-17T11:38:34.786ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
85732025-07-17T11:38:34.788ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
85742025-07-17T11:38:34.788ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 6.651508ms iteration = 154 status = {"chicken_switches_updated":false}
85752025-07-17T11:38:39.782ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 78 reason = Timeout
85762025-07-17T11:38:39.782ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 78 reason = Timeout
85772025-07-17T11:38:39.782ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
85782025-07-17T11:38:39.782ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
85792025-07-17T11:38:39.782ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 155 reason = Timeout
85802025-07-17T11:38:39.782ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
85812025-07-17T11:38:39.783ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
85822025-07-17T11:38:39.783ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
85832025-07-17T11:38:39.783ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
85842025-07-17T11:38:39.785ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
85852025-07-17T11:38:39.785ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
85862025-07-17T11:38:39.785ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85872025-07-17T11:38:39.786ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85882025-07-17T11:38:39.787ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.59205ms iteration = 78 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
85892025-07-17T11:38:39.788ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
85902025-07-17T11:38:39.788ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
85912025-07-17T11:38:39.788ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
85922025-07-17T11:38:39.789ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
85932025-07-17T11:38:39.789ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
85942025-07-17T11:38:39.789ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
85952025-07-17T11:38:39.791ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
85962025-07-17T11:38:39.791ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
85972025-07-17T11:38:39.791ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
85982025-07-17T11:38:39.791ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 9.419864ms iteration = 155 status = {"chicken_switches_updated":false}
85992025-07-17T11:38:39.792ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86002025-07-17T11:38:39.795ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
86012025-07-17T11:38:39.795ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
86022025-07-17T11:38:39.795ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
86032025-07-17T11:38:39.808ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
86042025-07-17T11:38:39.808ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 26.081164ms iteration = 78 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
86052025-07-17T11:38:41.694ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 375 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = 999b1d45-8ac7-48a4-b19b-3abea049f998 response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
86062025-07-17T11:38:44.782ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 156 reason = Timeout
86072025-07-17T11:38:44.782ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
86082025-07-17T11:38:44.783ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
86092025-07-17T11:38:44.785ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86102025-07-17T11:38:44.787ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
86112025-07-17T11:38:44.787ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
86122025-07-17T11:38:44.788ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
86132025-07-17T11:38:44.790ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
86142025-07-17T11:38:44.790ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 7.322596ms iteration = 156 status = {"chicken_switches_updated":false}
86152025-07-17T11:38:49.678ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 14 reason = Timeout
86162025-07-17T11:38:49.678ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 14 reason = Timeout
86172025-07-17T11:38:49.678ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
86182025-07-17T11:38:49.678ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 14 reason = Timeout
86192025-07-17T11:38:49.678ZINFOnexus (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-07-17 11:28:49.678656824 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:48
86202025-07-17T11:38:49.678ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
86212025-07-17T11:38:49.678ZDEBGnexus (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
86222025-07-17T11:38:49.678ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 16 reason = Timeout
86232025-07-17T11:38:49.678ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 14 reason = Timeout
86242025-07-17T11:38:49.678ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
86252025-07-17T11:38:49.678ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
86262025-07-17T11:38:49.678ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 16 reason = Timeout
86272025-07-17T11:38:49.678ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 27 reason = Timeout
86282025-07-17T11:38:49.678ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
86292025-07-17T11:38:49.678ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 14 reason = Timeout
86302025-07-17T11:38:49.678ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 14 reason = Timeout
86312025-07-17T11:38:49.678ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
86322025-07-17T11:38:49.679ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("3dd3bf3b-7760-4750-b1d3-52ee777335f0.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("79e5c77d-18b3-4951-8d11-cfa3eae2b3e3.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("79e5c77d-18b3-4951-8d11-cfa3eae2b3e3.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::8)) }], valid_until: Instant { tv_sec: 4123, tv_nsec: 429556536 } })
86332025-07-17T11:38:49.679ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
86342025-07-17T11:38:49.679ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
86352025-07-17T11:38:49.679ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
86362025-07-17T11:38:49.679ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
86372025-07-17T11:38:49.679ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("3144a75a-317f-4b68-93e2-a210ca1189d1.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("87408888-9df4-4e0c-bc8e-2f920d039100.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("ee4378c5-e5f2-41cf-8969-e5e03e1b9a41.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("ee4378c5-e5f2-41cf-8969-e5e03e1b9a41.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:3::1)) }], valid_until: Instant { tv_sec: 4123, tv_nsec: 429812819 } })
86382025-07-17T11:38:49.679ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
86392025-07-17T11:38:49.679ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
86402025-07-17T11:38:49.679ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 1.278367ms iteration = 14 status = {"addresses":["[fd00:1122:3344:101::9]:5353","[fd00:1122:3344:101::8]:5353"]}
86412025-07-17T11:38:49.680ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } dns_group = internal
86422025-07-17T11:38:49.680ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 1.333128ms iteration = 14 status = {"addresses":["[fd00:1122:3344:1::1]:5353","[fd00:1122:3344:2::1]:5353","[fd00:1122:3344:3::1]:5353"]}
86432025-07-17T11:38:49.681ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
86442025-07-17T11:38:49.681ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 15 reason = Timeout
86452025-07-17T11:38:49.681ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/tasks/blueprint_execution.rs:85 reason = blueprint disabled target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a
86462025-07-17T11:38:49.681ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 12.03\u{b5}s iteration = 15 status = {"enabled":false,"target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a"}
86472025-07-17T11:38:49.681ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
86482025-07-17T11:38:49.681ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86492025-07-17T11:38:49.681ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
86502025-07-17T11:38:49.681ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
86512025-07-17T11:38:49.683ZINFOnexus (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-07-17 11:28:49.678656824 UTC file = nexus/src/app/background/tasks/metrics_producer_gc.rs:69 pruned = {}
86522025-07-17T11:38:49.683ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 5.27177ms iteration = 14 status = {"expiration":"2025-07-17T11:28:49.678656824Z","pruned":[]}
86532025-07-17T11:38:49.683ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86542025-07-17T11:38:49.683ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 4123, tv_nsec: 434308838 } })
86552025-07-17T11:38:49.684ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
86562025-07-17T11:38:49.684ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
86572025-07-17T11:38:49.684ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
86582025-07-17T11:38:49.684ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1186
86592025-07-17T11:38:49.685ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86602025-07-17T11:38:49.687ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_planner iteration = 17 reason = Timeout
86612025-07-17T11:38:49.687ZDEBGnexus (BackgroundTasks): blueprint planning disabled, doing nothing background_task = blueprint_planner
86622025-07-17T11:38:49.687ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_planner elapsed = 12.56\u{b5}s iteration = 17 status = "Disabled"
86632025-07-17T11:38:49.687ZDEBGnexus (BackgroundTasks): activating background_task = instance_reincarnation iteration = 14 reason = Timeout
86642025-07-17T11:38:49.687ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
86652025-07-17T11:38:49.687ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
86662025-07-17T11:38:49.687ZDEBGnexus (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 14 reason = Timeout
86672025-07-17T11:38:49.687ZINFOnexus (BackgroundTasks): Decommissioned disk cleaner running background_task = decommissioned_disk_cleaner file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:53
86682025-07-17T11:38:49.687ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
86692025-07-17T11:38:49.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
86702025-07-17T11:38:49.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
86712025-07-17T11:38:49.687ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
86722025-07-17T11:38:49.688ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
86732025-07-17T11:38:49.688ZDEBGnexus (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 14 reason = Timeout
86742025-07-17T11:38:49.689ZDEBGnexus (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
86752025-07-17T11:38:49.689ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86762025-07-17T11:38:49.689ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
86772025-07-17T11:38:49.689ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
86782025-07-17T11:38:49.690ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86792025-07-17T11:38:49.690ZDEBGnexus (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
86802025-07-17T11:38:49.691ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
86812025-07-17T11:38:49.691ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
86822025-07-17T11:38:49.691ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
86832025-07-17T11:38:49.691ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
86842025-07-17T11:38:49.691ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
86852025-07-17T11:38:49.691ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
86862025-07-17T11:38:49.692ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86872025-07-17T11:38:49.693ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
86882025-07-17T11:38:49.693ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
86892025-07-17T11:38:49.693ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
86902025-07-17T11:38:49.694ZDEBGnexus (BackgroundTasks): activating background_task = lookup_region_port iteration = 14 reason = Timeout
86912025-07-17T11:38:49.694ZDEBGnexus (BackgroundTasks): activating background_task = webhook_deliverator iteration = 14 reason = Timeout
86922025-07-17T11:38:49.694ZDEBGnexus (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
86932025-07-17T11:38:49.694ZDEBGnexus (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} }
86942025-07-17T11:38:49.694ZDEBGnexus (BackgroundTasks): activating background_task = alert_dispatcher iteration = 14 reason = Timeout
86952025-07-17T11:38:49.694ZDEBGnexus (BackgroundTasks): roles background_task = alert_dispatcher roles = RoleSet { roles: {} }
86962025-07-17T11:38:49.694ZDEBGnexus (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
86972025-07-17T11:38:49.694ZINFOnexus (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper file = nexus/src/app/background/tasks/abandoned_vmm_reaper.rs:170 sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
86982025-07-17T11:38:49.694ZDEBGnexus (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 5.29893ms iteration = 14 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
86992025-07-17T11:38:49.694ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database result = Ok(())
87002025-07-17T11:38:49.694ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
87012025-07-17T11:38:49.694ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = alert_dispatcher resource = Database result = Ok(())
87022025-07-17T11:38:49.694ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Fleet result = Ok(())
87032025-07-17T11:38:49.694ZDEBGnexus (BackgroundTasks): roles background_task = decommissioned_disk_cleaner roles = RoleSet { roles: {} }
87042025-07-17T11:38:49.694ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = decommissioned_disk_cleaner resource = Database result = Ok(())
87052025-07-17T11:38:49.695ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
87062025-07-17T11:38:49.695ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
87072025-07-17T11:38:49.696ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
87082025-07-17T11:38:49.696ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
87092025-07-17T11:38:49.696ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
87102025-07-17T11:38:49.696ZINFOnexus (BackgroundTasks): all webhook delivery tasks completed background_task = webhook_deliverator file = nexus/src/app/background/tasks/webhook_deliverator.rs:204 num_receivers = 0
87112025-07-17T11:38:49.696ZDEBGnexus (BackgroundTasks): activation complete background_task = webhook_deliverator elapsed = 2.840458ms iteration = 14 status = {"by_rx":{},"error":null}
87122025-07-17T11:38:49.697ZDEBGnexus (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 2.973269ms iteration = 14 status = {"errors":[],"found_port_ok":[]}
87132025-07-17T11:38:49.697ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87142025-07-17T11:38:49.699ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
87152025-07-17T11:38:49.699ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
87162025-07-17T11:38:49.699ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
87172025-07-17T11:38:49.700ZDEBGnexus (BackgroundTasks): roles background_task = alert_dispatcher roles = RoleSet { roles: {} }
87182025-07-17T11:38:49.700ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = alert_dispatcher resource = Database result = Ok(())
87192025-07-17T11:38:49.702ZDEBGnexus (BackgroundTasks): activation complete background_task = alert_dispatcher elapsed = 8.661174ms iteration = 14 status = {"dispatched":[],"errors":[],"glob_version":"161.0.0","globs_reprocessed":{},"no_receivers":[]}
87202025-07-17T11:38:49.735ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 27 reason = Timeout
87212025-07-17T11:38:49.735ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 27 reason = Timeout
87222025-07-17T11:38:49.735ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
87232025-07-17T11:38:49.735ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
87242025-07-17T11:38:49.738ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
87252025-07-17T11:38:49.738ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
87262025-07-17T11:38:49.738ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
87272025-07-17T11:38:49.738ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
87282025-07-17T11:38:49.738ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
87292025-07-17T11:38:49.739ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.688369ms iteration = 27 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
87302025-07-17T11:38:49.739ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 4123, tv_nsec: 489480566 } })
87312025-07-17T11:38:49.739ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1186
87322025-07-17T11:38:49.743ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87332025-07-17T11:38:49.747ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
87342025-07-17T11:38:49.747ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
87352025-07-17T11:38:49.748ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
87362025-07-17T11:38:49.748ZDEBGnexus (BackgroundTasks): Found zpools on decommissioned disks background_task = decommissioned_disk_cleaner count = 0
87372025-07-17T11:38:49.748ZINFOnexus (BackgroundTasks): Cleaned decommissioned zpools background_task = decommissioned_disk_cleaner deleted = 0 error_count = 0 file = nexus/src/app/background/tasks/decommissioned_disk_cleaner.rs:138 found = 0 not_ready_to_be_deleted = 0
87382025-07-17T11:38:49.748ZDEBGnexus (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 61.219578ms iteration = 14 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
87392025-07-17T11:38:49.750ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
87402025-07-17T11:38:49.750ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 915fbba0-947f-4bf0-9f9f-48a79df81f7b dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
87412025-07-17T11:38:49.750ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
87422025-07-17T11:38:49.750ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
87432025-07-17T11:38:49.752ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87442025-07-17T11:38:49.756ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
87452025-07-17T11:38:49.756ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
87462025-07-17T11:38:49.756ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
87472025-07-17T11:38:49.761ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 915fbba0-947f-4bf0-9f9f-48a79df81f7b dns_zone_name = control-plane.oxide.internal found_so_far = 59 version = 1
87482025-07-17T11:38:49.761ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
87492025-07-17T11:38:49.761ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2025-07-17 11:26:34.922237 UTC dns_group = internal generation = 1
87502025-07-17T11:38:49.761ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 82.441447ms iteration = 14 status = {"generation":1}
87512025-07-17T11:38:49.764ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
87522025-07-17T11:38:49.764ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 4a9e4c5e-728e-4850-bd58-14548c02bf6f dns_zone_name = oxide.test found_so_far = 0 version = 2
87532025-07-17T11:38:49.764ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
87542025-07-17T11:38:49.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
87552025-07-17T11:38:49.768ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-07-17T11:25:46.964149Z, time_modified: 2025-07-17T11:25:46.964149Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 1fa836d9-7854-4ada-8c89-c26d922646c6, db_silo: Silo { identity: SiloIdentity { id: 1fa836d9-7854-4ada-8c89-c26d922646c6, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-07-17T11:26:35.643175Z, time_modified: 2025-07-17T11:26:35.643175Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "d7c3455b70a66c86ec2a7e597ca46fe0ea0c49dde5c1363f198c20aa77d58442" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 1fa836d9-7854-4ada-8c89-c26d922646c6, db_silo: Silo { identity: SiloIdentity { id: 1fa836d9-7854-4ada-8c89-c26d922646c6, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2025-07-17T11:26:35.643175Z, time_modified: 2025-07-17T11:26:35.643175Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "d7c3455b70a66c86ec2a7e597ca46fe0ea0c49dde5c1363f198c20aa77d58442" }] }) }
87562025-07-17T11:38:49.768ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 89.387799ms iteration = 14 status = {"by_dns_name":{"default-silo.sys.oxide.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"recovery.sys.oxide.test":{"silo_id":"1fa836d9-7854-4ada-8c89-c26d922646c6","tls_certs":["d7c3455b70a66c86ec2a7e597ca46fe0ea0c49dde5c1363f198c20aa77d58442"]}},"default_endpoint":{"silo_id":"1fa836d9-7854-4ada-8c89-c26d922646c6","tls_certs":["d7c3455b70a66c86ec2a7e597ca46fe0ea0c49dde5c1363f198c20aa77d58442"]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide.test\\" has no usable certificates"]}
87572025-07-17T11:38:49.768ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87582025-07-17T11:38:49.773ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
87592025-07-17T11:38:49.773ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
87602025-07-17T11:38:49.773ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
87612025-07-17T11:38:49.775ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 4a9e4c5e-728e-4850-bd58-14548c02bf6f dns_zone_name = oxide.test found_so_far = 4 version = 2
87622025-07-17T11:38:49.775ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
87632025-07-17T11:38:49.775ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2025-07-17 11:26:35.847594 UTC dns_group = external generation = 2
87642025-07-17T11:38:49.775ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 96.938768ms iteration = 14 status = {"generation":2}
87652025-07-17T11:38:49.782ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 27 reason = Timeout
87662025-07-17T11:38:49.782ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 27 reason = Timeout
87672025-07-17T11:38:49.782ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
87682025-07-17T11:38:49.782ZDEBGnexus (BackgroundTasks): activating background_task = v2p_manager iteration = 27 reason = Timeout
87692025-07-17T11:38:49.782ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
87702025-07-17T11:38:49.782ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 29 reason = Timeout
87712025-07-17T11:38:49.782ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
87722025-07-17T11:38:49.782ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
87732025-07-17T11:38:49.782ZDEBGnexus (BackgroundTasks): activating background_task = support_bundle_collector iteration = 27 reason = Timeout
87742025-07-17T11:38:49.782ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 79 reason = Timeout
87752025-07-17T11:38:49.782ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
87762025-07-17T11:38:49.782ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
87772025-07-17T11:38:49.782ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 27 reason = Timeout
87782025-07-17T11:38:49.782ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
87792025-07-17T11:38:49.782ZDEBGnexus (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 27 reason = Timeout
87802025-07-17T11:38:49.782ZDEBGnexus (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
87812025-07-17T11:38:49.782ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 79 reason = Timeout
87822025-07-17T11:38:49.782ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 27 reason = Timeout
87832025-07-17T11:38:49.782ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
87842025-07-17T11:38:49.782ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
87852025-07-17T11:38:49.782ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 157 reason = Timeout
87862025-07-17T11:38:49.782ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
87872025-07-17T11:38:49.782ZDEBGnexus (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 27 reason = Timeout
87882025-07-17T11:38:49.782ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
87892025-07-17T11:38:49.782ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 27 reason = Timeout
87902025-07-17T11:38:49.782ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
87912025-07-17T11:38:49.782ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 27 reason = Timeout
87922025-07-17T11:38:49.782ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
87932025-07-17T11:38:49.782ZDEBGnexus (BackgroundTasks): activating background_task = instance_updater iteration = 27 reason = Timeout
87942025-07-17T11:38:49.782ZDEBGnexus (BackgroundTasks): looking for instances with destroyed active VMMs... background_task = instance_updater
87952025-07-17T11:38:49.782ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
87962025-07-17T11:38:49.782ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 27 reason = Timeout
87972025-07-17T11:38:49.782ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
87982025-07-17T11:38:49.782ZDEBGnexus (BackgroundTasks): activating background_task = sp_ereport_ingester iteration = 27 reason = Timeout
87992025-07-17T11:38:49.782ZDEBGnexus (BackgroundTasks): activating background_task = vpc_route_manager iteration = 27 reason = Timeout
88002025-07-17T11:38:49.782ZINFOnexus (BackgroundTasks): VPC route manager running background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:60
88012025-07-17T11:38:49.782ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
88022025-07-17T11:38:49.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
88032025-07-17T11:38:49.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
88042025-07-17T11:38:49.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
88052025-07-17T11:38:49.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
88062025-07-17T11:38:49.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
88072025-07-17T11:38:49.783ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
88082025-07-17T11:38:49.783ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
88092025-07-17T11:38:49.783ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
88102025-07-17T11:38:49.783ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
88112025-07-17T11:38:49.783ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
88122025-07-17T11:38:49.783ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
88132025-07-17T11:38:49.783ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
88142025-07-17T11:38:49.783ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
88152025-07-17T11:38:49.783ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
88162025-07-17T11:38:49.783ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
88172025-07-17T11:38:49.783ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
88182025-07-17T11:38:49.783ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
88192025-07-17T11:38:49.783ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12225, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 4123, tv_nsec: 533587837 } })
88202025-07-17T11:38:49.786ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88212025-07-17T11:38:49.787ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
88222025-07-17T11:38:49.787ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
88232025-07-17T11:38:49.788ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88242025-07-17T11:38:49.789ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
88252025-07-17T11:38:49.789ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
88262025-07-17T11:38:49.789ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
88272025-07-17T11:38:49.789ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88282025-07-17T11:38:49.790ZDEBGnexus (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 7.741152ms iteration = 27 status = {"errors":[],"requests_created_ok":[]}
88292025-07-17T11:38:49.790ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88302025-07-17T11:38:49.791ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88312025-07-17T11:38:49.792ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
88322025-07-17T11:38:49.792ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
88332025-07-17T11:38:49.792ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
88342025-07-17T11:38:49.792ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88352025-07-17T11:38:49.793ZINFOnexus (BackgroundTasks): listed instances with destroyed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
88362025-07-17T11:38:49.793ZDEBGnexus (BackgroundTasks): looking for instances with failed active VMMs... background_task = instance_updater
88372025-07-17T11:38:49.793ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
88382025-07-17T11:38:49.793ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88392025-07-17T11:38:49.793ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
88402025-07-17T11:38:49.794ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
88412025-07-17T11:38:49.794ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
88422025-07-17T11:38:49.794ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88432025-07-17T11:38:49.795ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
88442025-07-17T11:38:49.795ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
88452025-07-17T11:38:49.795ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
88462025-07-17T11:38:49.795ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
88472025-07-17T11:38:49.795ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
88482025-07-17T11:38:49.796ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
88492025-07-17T11:38:49.797ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
88502025-07-17T11:38:49.797ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
88512025-07-17T11:38:49.797ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
88522025-07-17T11:38:49.798ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88532025-07-17T11:38:49.798ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("dendrite-0ae78f24-48db-4cf7-91cb-8d5311f2a17f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 4123, tv_nsec: 549176503 } })
88542025-07-17T11:38:49.799ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
88552025-07-17T11:38:49.799ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
88562025-07-17T11:38:49.799ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
88572025-07-17T11:38:49.799ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
88582025-07-17T11:38:49.799ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1186 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
88592025-07-17T11:38:49.799ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
88602025-07-17T11:38:49.799ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
88612025-07-17T11:38:49.799ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
88622025-07-17T11:38:49.799ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
88632025-07-17T11:38:49.799ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
88642025-07-17T11:38:49.800ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
88652025-07-17T11:38:49.800ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
88662025-07-17T11:38:49.801ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
88672025-07-17T11:38:49.801ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
88682025-07-17T11:38:49.802ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
88692025-07-17T11:38:49.803ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
88702025-07-17T11:38:49.803ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 20.704653ms iteration = 157 status = {"chicken_switches_updated":false}
88712025-07-17T11:38:49.804ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88722025-07-17T11:38:49.808ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
88732025-07-17T11:38:49.808ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
88742025-07-17T11:38:49.808ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
88752025-07-17T11:38:49.812ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
88762025-07-17T11:38:49.812ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
88772025-07-17T11:38:49.818ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
88782025-07-17T11:38:49.824ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
88792025-07-17T11:38:49.831ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
88802025-07-17T11:38:49.831ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 49.004036ms iteration = 79 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
88812025-07-17T11:38:49.836ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1194 zone_address = fd00:1122:3344:101::2
88822025-07-17T11:38:49.836ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
88832025-07-17T11:38:49.836ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "71e63880-693e-45ea-803d-b825cdc22f6a", "content-length": "26", "date": "Thu, 17 Jul 2025 11:38:49 GMT"} })
88842025-07-17T11:38:49.836ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1197 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
88852025-07-17T11:38:49.854ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
88862025-07-17T11:38:49.854ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
88872025-07-17T11:38:49.876ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
88882025-07-17T11:38:49.884ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 101.65826ms iteration = 79 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
88892025-07-17T11:38:49.892ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 109.705027ms iteration = 27 status = {"errors":[],"garbage_collect_requested":[]}
88902025-07-17T11:38:49.893ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 111.229436ms iteration = 27 status = {"errors":[],"finish_invoked_ok":[]}
88912025-07-17T11:38:49.900ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
88922025-07-17T11:38:49.900ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
88932025-07-17T11:38:49.901ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
88942025-07-17T11:38:49.901ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/tasks/instance_watcher.rs:550 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
88952025-07-17T11:38:49.901ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 118.595114ms iteration = 27 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
88962025-07-17T11:38:49.903ZINFOnexus (BackgroundTasks): listed instances with failed active VMMs background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
88972025-07-17T11:38:49.903ZDEBGnexus (BackgroundTasks): looking for instances with terminated active migrations... background_task = instance_updater
88982025-07-17T11:38:49.903ZDEBGnexus (BackgroundTasks): roles background_task = instance_updater roles = RoleSet { roles: {} }
88992025-07-17T11:38:49.904ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_updater resource = Database result = Ok(())
89002025-07-17T11:38:49.904ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
89012025-07-17T11:38:49.905ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
89022025-07-17T11:38:49.906ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
89032025-07-17T11:38:49.907ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89042025-07-17T11:38:49.909ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 126.736821ms iteration = 27 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
89052025-07-17T11:38:49.909ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 126.703481ms iteration = 27 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
89062025-07-17T11:38:49.909ZINFOnexus (BackgroundTasks): instance reincarnation completed successfully background_task = instance_reincarnation file = nexus/src/app/background/tasks/instance_reincarnation.rs:113 instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0
89072025-07-17T11:38:49.909ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 221.839135ms iteration = 14 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
89082025-07-17T11:38:49.910ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
89092025-07-17T11:38:49.910ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
89102025-07-17T11:38:49.910ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
89112025-07-17T11:38:49.927ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
89122025-07-17T11:38:49.928ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
89132025-07-17T11:38:49.930ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
89142025-07-17T11:38:49.931ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
89152025-07-17T11:38:49.934ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89162025-07-17T11:38:49.938ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 0ae78f24-48db-4cf7-91cb-8d5311f2a17f, lookup_type: ById(0ae78f24-48db-4cf7-91cb-8d5311f2a17f) } result = Ok(())
89172025-07-17T11:38:49.938ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
89182025-07-17T11:38:49.939ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
89192025-07-17T11:38:49.942ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 159.568504ms iteration = 27 status = 0
89202025-07-17T11:38:49.972ZDEBGnexus (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
89212025-07-17T11:38:49.972ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
89222025-07-17T11:38:49.974ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89232025-07-17T11:38:49.974ZDEBGnexus (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 192.585769ms iteration = 27 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
89242025-07-17T11:38:49.978ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
89252025-07-17T11:38:49.978ZDEBGnexus (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
89262025-07-17T11:38:49.978ZINFOnexus (BackgroundTasks): listed instances with terminated active migrations background_task = instance_updater count = 0 file = nexus/src/app/background/tasks/instance_updater.rs:59
89272025-07-17T11:38:49.978ZINFOnexus (BackgroundTasks): instance updater activation completed background_task = instance_updater destroyed_active_vmms = 0 failed_active_vmms = 0 file = nexus/src/app/background/tasks/instance_updater.rs:225 terminated_active_migrations = 0 update_sagas_completed = 0 update_sagas_started = 0
89282025-07-17T11:38:49.978ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_updater elapsed = 195.63427ms iteration = 27 status = {"destroyed_active_vmms":0,"disabled":false,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
89292025-07-17T11:38:49.978ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
89302025-07-17T11:38:49.980ZINFOnexus (BackgroundTasks): No bundles to collect background_task = support_bundle_collector file = nexus/src/app/background/tasks/support_bundle_collector.rs:372
89312025-07-17T11:38:49.980ZDEBGnexus (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 198.074281ms iteration = 27 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
89322025-07-17T11:38:49.985ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
89332025-07-17T11:38:49.985ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
89342025-07-17T11:38:49.995ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
89352025-07-17T11:38:49.996ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
89362025-07-17T11:38:49.998ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89372025-07-17T11:38:50.000ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
89382025-07-17T11:38:50.000ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
89392025-07-17T11:38:50.000ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
89402025-07-17T11:38:50.001ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89412025-07-17T11:38:50.003ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
89422025-07-17T11:38:50.003ZDEBGnexus (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
89432025-07-17T11:38:50.003ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
89442025-07-17T11:38:50.049ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 267.03502ms iteration = 29 status = {"physical_disks_added":0}
89452025-07-17T11:38:50.061ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1194 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 zone_address = fd00:1122:3344:101::2
89462025-07-17T11:38:50.061ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
89472025-07-17T11:38:50.062ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3488abb-158f-4996-9458-103044694265", "content-length": "26", "date": "Thu, 17 Jul 2025 11:38:49 GMT"} })
89482025-07-17T11:38:50.062ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1197 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
89492025-07-17T11:38:50.064ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
89502025-07-17T11:38:50.123ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
89512025-07-17T11:38:50.123ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::9]:5353/config", status: 204, headers: {"x-request-id": "c949a1c2-597e-44c9-9b80-b588abbb749f", "date": "Thu, 17 Jul 2025 11:38:50 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
89522025-07-17T11:38:50.123ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
89532025-07-17T11:38:50.125ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: "http://[fd00:1122:3344:101::8]:5353/config", status: 204, headers: {"x-request-id": "59668180-f038-4abc-86eb-63bd0a0bc3f7", "date": "Thu, 17 Jul 2025 11:38:50 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
89542025-07-17T11:38:50.125ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
89552025-07-17T11:38:50.128ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
89562025-07-17T11:38:50.128ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 449.532926ms iteration = 16 status = {"generation":2,"server_results":{"[fd00:1122:3344:101::8]:5353":{"Ok":null},"[fd00:1122:3344:101::9]:5353":{"Ok":null}}}
89572025-07-17T11:38:50.131ZINFOnexus (BackgroundTasks): VPC route manager sled 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:98
89582025-07-17T11:38:50.131ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
89592025-07-17T11:38:50.131ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "c700666d-bfa2-428d-bf6b-60ff3fe3d5a0", "content-length": "372", "date": "Thu, 17 Jul 2025 11:38:50 GMT"} })
89602025-07-17T11:38:50.131ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
89612025-07-17T11:38:50.132ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
89622025-07-17T11:38:50.132ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
89632025-07-17T11:38:50.147ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1194 zone_address = fd00:1122:3344:101::2
89642025-07-17T11:38:50.147ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
89652025-07-17T11:38:50.148ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "492f2136-245f-418f-b9b5-43b828eaea3c", "content-length": "26", "date": "Thu, 17 Jul 2025 11:38:49 GMT"} })
89662025-07-17T11:38:50.148ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1197 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
89672025-07-17T11:38:50.149ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1231 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
89682025-07-17T11:38:50.181ZINFOnexus (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:116 sled = sock
89692025-07-17T11:38:50.181ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/eip-gateways
89702025-07-17T11:38:50.182ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/eip-gateways", status: 204, headers: {"x-request-id": "1a1e839e-c36d-4857-9f05-53de5ce59652", "date": "Thu, 17 Jul 2025 11:38:50 GMT"} })
89712025-07-17T11:38:50.182ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
89722025-07-17T11:38:50.182ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
89732025-07-17T11:38:50.184ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
89742025-07-17T11:38:50.185ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
89752025-07-17T11:38:50.188ZDEBGnexus (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
89762025-07-17T11:38:50.188ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
89772025-07-17T11:38:50.192ZINFOnexus (BackgroundTasks): VPC route manager sled 0ae78f24-48db-4cf7-91cb-8d5311f2a17f push not needed background_task = vpc_route_manager file = nexus/src/app/background/tasks/vpc_routes.rs:262
89782025-07-17T11:38:50.192ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc-routes
89792025-07-17T11:38:50.192ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = vpc_route_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/vpc-routes", status: 204, headers: {"x-request-id": "6d4c4089-5fc0-4e19-8692-cc9fa46313b9", "date": "Thu, 17 Jul 2025 11:38:50 GMT"} })
89802025-07-17T11:38:50.194ZDEBGnexus (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 411.785699ms iteration = 27 status = {}
89812025-07-17T11:38:50.222ZDEBGnexus (BackgroundTasks): client request SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = v2p_manager body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/v2p
89822025-07-17T11:38:50.222ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
89832025-07-17T11:38:50.222ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
89842025-07-17T11:38:50.222ZDEBGnexus (BackgroundTasks): client response SledAgent = 0ae78f24-48db-4cf7-91cb-8d5311f2a17f background_task = v2p_manager result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c07bd94-9755-4905-90e6-0e248290f792", "content-length": "691", "date": "Thu, 17 Jul 2025 11:38:50 GMT"} })
89852025-07-17T11:38:50.222ZINFOnexus (BackgroundTasks): found opte v2p mappings background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:102 interfaces = {} sled = sock
89862025-07-17T11:38:50.222ZINFOnexus (BackgroundTasks): v2p mappings to delete background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:114 mappings = [] sled = sock
89872025-07-17T11:38:50.222ZINFOnexus (BackgroundTasks): v2p mappings to add background_task = v2p_manager file = nexus/src/app/background/tasks/v2p_mappings.rs:127 mappings = [] sled = sock
89882025-07-17T11:38:50.224ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 545.654083ms iteration = 27 status = null
89892025-07-17T11:38:50.225ZDEBGnexus (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 442.798918ms iteration = 27 status = {}
89902025-07-17T11:38:50.226ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/all-sp-ids
89912025-07-17T11:38:50.227ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/local/all-sp-ids", status: 200, headers: {"content-type": "application/json", "x-request-id": "36c621b3-ca74-462d-8895-61ce79c345d2", "content-length": "78", "date": "Thu, 17 Jul 2025 11:38:49 GMT"} })
89922025-07-17T11:38:50.227ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 1 sp_type = sled
89932025-07-17T11:38:50.227ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = switch
89942025-07-17T11:38:50.227ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = sled
89952025-07-17T11:38:50.227ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 1 sp_type = sled
89962025-07-17T11:38:50.227ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = switch
89972025-07-17T11:38:50.227ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = sled
89982025-07-17T11:38:50.229ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 0 sp_type = switch
89992025-07-17T11:38:50.229ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 1 sp_type = sled
90002025-07-17T11:38:50.229ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } slot = 0 sp_type = sled
90012025-07-17T11:38:50.232ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 0 sp_type = switch
90022025-07-17T11:38:50.232ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = switch
90032025-07-17T11:38:50.232ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 1 sp_type = sled
90042025-07-17T11:38:50.232ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 1 sp_type = sled
90052025-07-17T11:38:50.232ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = switch
90062025-07-17T11:38:50.232ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 1 sp_type = sled
90072025-07-17T11:38:50.232ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Fleet result = Ok(()) slot = 0 sp_type = sled
90082025-07-17T11:38:50.232ZDEBGnexus (BackgroundTasks): roles background_task = sp_ereport_ingester roles = RoleSet { roles: {} } slot = 0 sp_type = sled
90092025-07-17T11:38:50.233ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = sp_ereport_ingester resource = Database result = Ok(()) slot = 0 sp_type = sled
90102025-07-17T11:38:50.264ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
90112025-07-17T11:38:50.265ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:1::1]:5353/config", status: 204, headers: {"x-request-id": "faaf6a44-c443-421b-94a5-c904cda6a065", "date": "Thu, 17 Jul 2025 11:38:50 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
90122025-07-17T11:38:50.265ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
90132025-07-17T11:38:50.268ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 532.771004ms iteration = 27 status = {}
90142025-07-17T11:38:50.268ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:2::1]:5353/config", status: 204, headers: {"x-request-id": "41d5e59b-54d0-419b-8bd9-c69a17920335", "date": "Thu, 17 Jul 2025 11:38:50 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
90152025-07-17T11:38:50.268ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
90162025-07-17T11:38:50.268ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 1 sp_type = sled start_ena = None
90172025-07-17T11:38:50.268ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/1/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
90182025-07-17T11:38:50.269ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 0 sp_type = switch start_ena = None
90192025-07-17T11:38:50.269ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
90202025-07-17T11:38:50.269ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/1/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "4622c141-3beb-4a38-ba7b-e27240f7852b", "content-length": "81", "date": "Thu, 17 Jul 2025 11:38:49 GMT"} })
90212025-07-17T11:38:50.269ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f56ed2a-eb76-40df-aa6d-b5231927eadc", "content-length": "81", "date": "Thu, 17 Jul 2025 11:38:49 GMT"} })
90222025-07-17T11:38:50.270ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: "http://[fd00:1122:3344:3::1]:5353/config", status: 204, headers: {"x-request-id": "fb65dace-3808-46bd-925b-aad088f0da9f", "date": "Thu, 17 Jul 2025 11:38:50 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
90232025-07-17T11:38:50.270ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/tasks/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
90242025-07-17T11:38:50.272ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/tasks/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:1::1]:5353, [fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353] }
90252025-07-17T11:38:50.272ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 594.131473ms iteration = 16 status = {"generation":1,"server_results":{"[fd00:1122:3344:1::1]:5353":{"Ok":null},"[fd00:1122:3344:2::1]:5353":{"Ok":null},"[fd00:1122:3344:3::1]:5353":{"Ok":null}}}
90262025-07-17T11:38:50.273ZDEBGnexus (BackgroundTasks): attempting ereport collection from MGS background_task = sp_ereport_ingester committed_ena = None gateway_addr = [fd00:1122:3344:101::2]:12225 restart_id = 00000000-0000-0000-0000-000000000000 (ereporter_restart) slot = 0 sp_type = sled start_ena = None
90272025-07-17T11:38:50.273ZDEBGnexus (BackgroundTasks): client request background_task = sp_ereport_ingester body = None gateway_url = http://[fd00:1122:3344:101::2]:12225 method = POST uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000
90282025-07-17T11:38:50.273ZDEBGnexus (BackgroundTasks): client response background_task = sp_ereport_ingester gateway_url = http://[fd00:1122:3344:101::2]:12225 result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/0/ereports?limit=255&restart_id=00000000-0000-0000-0000-000000000000", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c3870fd-b550-405f-9eb5-724e658fd8e5", "content-length": "81", "date": "Thu, 17 Jul 2025 11:38:49 GMT"} })
90292025-07-17T11:38:50.275ZDEBGnexus (BackgroundTasks): activation complete background_task = sp_ereport_ingester elapsed = 492.796127ms iteration = 27 status = {"errors":[],"sps":[]}
90302025-07-17T11:38:50.321ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
90312025-07-17T11:38:50.321ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
90322025-07-17T11:38:50.324ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:150 port = SwitchPort { id: 151caa7c-6a05-4995-94b4-7946d428117d, rack_id: 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5, switch_location: "switch0", port_name: Name(Name("qsfp0")), port_settings_id: Some(2e30c7ae-26da-46e4-9127-aac194921f20) } rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0
90332025-07-17T11:38:50.324ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
90342025-07-17T11:38:50.324ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
90352025-07-17T11:38:50.337ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1832 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.2.174], params: LinkCreate { autoneg: false, fec: Some(None), kr: false, lane: Some(LinkId(0)), speed: Speed40G, tx_eq: None } }} } switch_port_id = Name(Name("qsfp0"))
90362025-07-17T11:38:50.337ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1859 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0 switch_port_id = Name(Name("qsfp0")) switch_port_settings_id = 2e30c7ae-26da-46e4-9127-aac194921f20
90372025-07-17T11:38:50.337ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:378 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, Some(1))}}
90382025-07-17T11:38:50.337ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:382 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: {(10.151.2.100, Prefix4 { length: 0, value: 0.0.0.0 }, None, None)}}
90392025-07-17T11:38:50.337ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:391 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
90402025-07-17T11:38:50.337ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:397 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
90412025-07-17T11:38:50.337ZINFOnexus (BackgroundTasks): deleting static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:402 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
90422025-07-17T11:38:50.337ZINFOnexus (BackgroundTasks): removing static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2006 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 request = DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
90432025-07-17T11:38:50.337ZINFOnexus (BackgroundTasks): adding static routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:408 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 routes = {Switch0: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } }}
90442025-07-17T11:38:50.337ZINFOnexus (BackgroundTasks): adding static v4 routes background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2042 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 request = AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 10.151.2.100, prefix: Prefix4 { length: 0, value: 0.0.0.0 }, rib_priority: 1, vlan_id: None }] } } switch_location = Switch0
90452025-07-17T11:38:50.337ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:417 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5
90462025-07-17T11:38:50.338ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
90472025-07-17T11:38:50.338ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
90482025-07-17T11:38:50.340ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], lldp: Some(LldpPortConfig { chassis_id: None, management_addrs: None, port_description: None, port_id: None, status: Disabled, system_description: None, system_name: None }), port: "qsfp0", tx_eq: None }] file = nexus/src/app/background/tasks/sync_switch_configuration.rs:466 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch_location = Switch0
90492025-07-17T11:38:50.340ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
90502025-07-17T11:38:50.562ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/switch-ports", status: 204, headers: {"x-request-id": "4b4e66c8-55ae-48e0-9512-6b4129134bd3", "date": "Thu, 17 Jul 2025 11:38:50 GMT"} })
90512025-07-17T11:38:50.562ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/tasks/sync_switch_configuration.rs:769 rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 switch = Switch0
90522025-07-17T11:38:50.562ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
90532025-07-17T11:38:50.562ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 1d46a41f-ae2f-4e3b-b6d4-1fd6857db5e5 result = Ok(Response { url: "http://[fd00:1122:3344:101::1]:12345/network-bootstore-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "9070ca9d-0ac8-4633-8670-623b502866c6", "content-length": "682", "date": "Thu, 17 Jul 2025 11:38:50 GMT"} })
90542025-07-17T11:38:50.563ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
90552025-07-17T11:38:50.563ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
90562025-07-17T11:38:50.567ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
90572025-07-17T11:38:50.567ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
90582025-07-17T11:38:50.571ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
90592025-07-17T11:38:50.571ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
90602025-07-17T11:38:50.573ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
90612025-07-17T11:38:50.573ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
90622025-07-17T11:38:50.580ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 797.96381ms iteration = 27 status = {}
90632025-07-17T11:38:51.695ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 386 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = ef56f487-a7b1-4b53-86b1-cc8813e38dc7 response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
90642025-07-17T11:38:54.782ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 158 reason = Timeout
90652025-07-17T11:38:54.782ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
90662025-07-17T11:38:54.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
90672025-07-17T11:38:54.784ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90682025-07-17T11:38:54.787ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
90692025-07-17T11:38:54.787ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
90702025-07-17T11:38:54.787ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
90712025-07-17T11:38:54.789ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
90722025-07-17T11:38:54.789ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 7.507709ms iteration = 158 status = {"chicken_switches_updated":false}
90732025-07-17T11:38:59.782ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 80 reason = Timeout
90742025-07-17T11:38:59.782ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 80 reason = Timeout
90752025-07-17T11:38:59.782ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
90762025-07-17T11:38:59.782ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
90772025-07-17T11:38:59.782ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 159 reason = Timeout
90782025-07-17T11:38:59.782ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
90792025-07-17T11:38:59.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
90802025-07-17T11:38:59.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
90812025-07-17T11:38:59.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
90822025-07-17T11:38:59.785ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
90832025-07-17T11:38:59.785ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
90842025-07-17T11:38:59.785ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90852025-07-17T11:38:59.785ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90862025-07-17T11:38:59.787ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 5.480452ms iteration = 80 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
90872025-07-17T11:38:59.789ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
90882025-07-17T11:38:59.789ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
90892025-07-17T11:38:59.789ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
90902025-07-17T11:38:59.789ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
90912025-07-17T11:38:59.789ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
90922025-07-17T11:38:59.789ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
90932025-07-17T11:38:59.790ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
90942025-07-17T11:38:59.791ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
90952025-07-17T11:38:59.791ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
90962025-07-17T11:38:59.791ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 9.580796ms iteration = 159 status = {"chicken_switches_updated":false}
90972025-07-17T11:38:59.792ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90982025-07-17T11:38:59.795ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
90992025-07-17T11:38:59.795ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
91002025-07-17T11:38:59.795ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
91012025-07-17T11:38:59.807ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
91022025-07-17T11:38:59.807ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 25.088291ms iteration = 80 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
91032025-07-17T11:39:01.694ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 328 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = 86e6ae87-61d7-4fca-99aa-c1a2fa9e5874 response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
91042025-07-17T11:39:04.782ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 160 reason = Timeout
91052025-07-17T11:39:04.782ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
91062025-07-17T11:39:04.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
91072025-07-17T11:39:04.784ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91082025-07-17T11:39:04.787ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
91092025-07-17T11:39:04.787ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
91102025-07-17T11:39:04.788ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
91112025-07-17T11:39:04.789ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
91122025-07-17T11:39:04.789ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 7.455049ms iteration = 160 status = {"chicken_switches_updated":false}
91132025-07-17T11:39:09.781ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement_driver iteration = 81 reason = Timeout
91142025-07-17T11:39:09.782ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
91152025-07-17T11:39:09.782ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 81 reason = Timeout
91162025-07-17T11:39:09.782ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 161 reason = Timeout
91172025-07-17T11:39:09.782ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
91182025-07-17T11:39:09.782ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
91192025-07-17T11:39:09.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
91202025-07-17T11:39:09.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
91212025-07-17T11:39:09.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
91222025-07-17T11:39:09.784ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91232025-07-17T11:39:09.784ZDEBGnexus (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
91242025-07-17T11:39:09.784ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
91252025-07-17T11:39:09.785ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91262025-07-17T11:39:09.786ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 4.890384ms iteration = 81 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
91272025-07-17T11:39:09.787ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
91282025-07-17T11:39:09.787ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
91292025-07-17T11:39:09.787ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
91302025-07-17T11:39:09.788ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
91312025-07-17T11:39:09.788ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
91322025-07-17T11:39:09.788ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
91332025-07-17T11:39:09.789ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
91342025-07-17T11:39:09.789ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
91352025-07-17T11:39:09.790ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
91362025-07-17T11:39:09.790ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 8.31125ms iteration = 161 status = {"chicken_switches_updated":false}
91372025-07-17T11:39:09.790ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91382025-07-17T11:39:09.793ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a, lookup_type: ById(653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a) } result = Ok(())
91392025-07-17T11:39:09.793ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
91402025-07-17T11:39:09.793ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
91412025-07-17T11:39:09.802ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a original_time_created = 2025-07-17 11:25:38.406797 UTC target_id = 653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a time_created = 2025-07-17 11:25:38.406797 UTC
91422025-07-17T11:39:09.802ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.50384ms iteration = 81 status = {"enabled":false,"status":"target blueprint unchanged","target_id":"653d4f0b-e9eb-44b0-b22c-13d8d1d68f6a","time_created":"2025-07-17T11:25:38.406797Z"}
91432025-07-17T11:39:11.695ZINFOnexus (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 510 local_addr = [fd00:1122:3344:101::b]:63089 method = GET remote_addr = [fd00:1122:3344:101::d]:55268 req_id = 7d0c1af9-3aba-4318-bc44-3c44e403959d response_code = 200 uri = /5e9f3641-e36d-4c7a-ab2f-ad78baee50f9
91442025-07-17T11:39:14.781ZDEBGnexus (BackgroundTasks): activating background_task = chicken_switches_watcher iteration = 162 reason = Timeout
91452025-07-17T11:39:14.782ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
91462025-07-17T11:39:14.782ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
91472025-07-17T11:39:14.784ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91482025-07-17T11:39:14.787ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = BlueprintConfig result = Ok(())
91492025-07-17T11:39:14.787ZDEBGnexus (BackgroundTasks): roles background_task = chicken_switches_watcher roles = RoleSet { roles: {} }
91502025-07-17T11:39:14.787ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = chicken_switches_watcher resource = Database result = Ok(())
91512025-07-17T11:39:14.789ZDEBGnexus (BackgroundTasks): chicken switches load complete background_task = chicken_switches_watcher
91522025-07-17T11:39:14.789ZDEBGnexus (BackgroundTasks): activation complete background_task = chicken_switches_watcher elapsed = 7.206955ms iteration = 162 status = {"chicken_switches_updated":false}
91532025-07-17T11:39:16.686ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:1025 local_addr = [fd00:1122:3344:101::b]:12221 remote_addr = [fd00:1122:3344:101::d]:49985
91542025-07-17T11:39:16.686ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49985 req_id = 2c8a2516-c17b-4422-bbe7-b21c6a7baf1d roles = RoleSet { roles: {} } uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?sort_by=id_ascending
91552025-07-17T11:39:16.687ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49985 req_id = 2c8a2516-c17b-4422-bbe7-b21c6a7baf1d resource = Database result = Ok(()) uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?sort_by=id_ascending
91562025-07-17T11:39:16.689ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 3397 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49985 req_id = 2c8a2516-c17b-4422-bbe7-b21c6a7baf1d response_code = 200 uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?sort_by=id_ascending
91572025-07-17T11:39:16.690ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49985 req_id = 48467cb4-a2f5-4235-989e-efaf575dc4a2 roles = RoleSet { roles: {} } uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmMzlkM2U3MC02NmU0LTQ2OGEtYTQ0ZC03NDZkMjJiZWEzYTAifX0%3D
91582025-07-17T11:39:16.690ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49985 req_id = 48467cb4-a2f5-4235-989e-efaf575dc4a2 resource = Database result = Ok(()) uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmMzlkM2U3MC02NmU0LTQ2OGEtYTQ0ZC03NDZkMjJiZWEzYTAifX0%3D
91592025-07-17T11:39:16.693ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:867 latency_us = 2891 local_addr = [fd00:1122:3344:101::b]:12221 method = GET remote_addr = [fd00:1122:3344:101::d]:49985 req_id = 48467cb4-a2f5-4235-989e-efaf575dc4a2 response_code = 200 uri = /metrics/collectors/3fb899b3-2d86-4b00-b1d9-dcfa5f97d13d/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJmMzlkM2U3MC02NmU0LTQ2OGEtYTQ0ZC03NDZkMjJiZWEzYTAifX0%3D