Max level shown:
12024-05-15T03:45:04.210ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 20 reason = Timeout
22024-05-15T03:45:04.210ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
32024-05-15T03:45:04.210ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
42024-05-15T03:45:04.211ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52024-05-15T03:45:04.214ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
62024-05-15T03:45:04.214ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
72024-05-15T03:45:04.214ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
82024-05-15T03:45:04.216ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
92024-05-15T03:45:04.216ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
102024-05-15T03:45:04.217ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
112024-05-15T03:45:04.220ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0b55a4be-1fdc-4b91-a3a7-f964eab4522f, lookup_type: ById(0b55a4be-1fdc-4b91-a3a7-f964eab4522f) } result = Ok(())
122024-05-15T03:45:04.220ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
132024-05-15T03:45:04.220ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
142024-05-15T03:45:04.228ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f original_time_created = 2024-05-15 03:41:43.607350 UTC target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f time_created = 2024-05-15 03:41:43.607350 UTC
152024-05-15T03:45:04.228ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 17.880653ms iteration = 20
162024-05-15T03:45:04.732ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 155 local_addr = [fd00:1122:3344:101::c]:55127 method = GET remote_addr = [fd00:1122:3344:101::d]:37120 req_id = cbdeb41f-6938-404d-9b0e-2694445d1758 response_code = 200 uri = //86c56a1d-af69-43b6-8206-e5d1baefadb2
172024-05-15T03:45:14.210ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 21 reason = Timeout
182024-05-15T03:45:14.210ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
192024-05-15T03:45:14.210ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
202024-05-15T03:45:14.212ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
212024-05-15T03:45:14.214ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
222024-05-15T03:45:14.214ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
232024-05-15T03:45:14.214ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
242024-05-15T03:45:14.216ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
252024-05-15T03:45:14.216ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
262024-05-15T03:45:14.217ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
272024-05-15T03:45:14.219ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0b55a4be-1fdc-4b91-a3a7-f964eab4522f, lookup_type: ById(0b55a4be-1fdc-4b91-a3a7-f964eab4522f) } result = Ok(())
282024-05-15T03:45:14.219ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
292024-05-15T03:45:14.220ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
302024-05-15T03:45:14.226ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f original_time_created = 2024-05-15 03:41:43.607350 UTC target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f time_created = 2024-05-15 03:41:43.607350 UTC
312024-05-15T03:45:14.226ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 16.041723ms iteration = 21
322024-05-15T03:45:14.732ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 161 local_addr = [fd00:1122:3344:101::c]:55127 method = GET remote_addr = [fd00:1122:3344:101::d]:37120 req_id = 21db19b1-77a5-41c6-b933-acc5734f7f4e response_code = 200 uri = //86c56a1d-af69-43b6-8206-e5d1baefadb2
332024-05-15T03:45:24.210ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 22 reason = Timeout
342024-05-15T03:45:24.210ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
352024-05-15T03:45:24.210ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 8 reason = Timeout
362024-05-15T03:45:24.210ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
372024-05-15T03:45:24.210ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 8 reason = Timeout
382024-05-15T03:45:24.210ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
392024-05-15T03:45:24.210ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 8 reason = Timeout
402024-05-15T03:45:24.210ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:46
412024-05-15T03:45:24.210ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 8 reason = Timeout
422024-05-15T03:45:24.210ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 8 reason = Timeout
432024-05-15T03:45:24.210ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
442024-05-15T03:45:24.210ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
452024-05-15T03:45:24.210ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
462024-05-15T03:45:24.210ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 8 reason = Timeout
472024-05-15T03:45:24.210ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 8 reason = Timeout
482024-05-15T03:45:24.210ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:44
492024-05-15T03:45:24.210ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:48
502024-05-15T03:45:24.210ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
512024-05-15T03:45:24.210ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 9.81\u{b5}s iteration = 8
522024-05-15T03:45:24.210ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
532024-05-15T03:45:24.210ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
542024-05-15T03:45:24.211ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
552024-05-15T03:45:24.211ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
562024-05-15T03:45:24.211ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
572024-05-15T03:45:24.211ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 8 reason = Timeout
582024-05-15T03:45:24.211ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
592024-05-15T03:45:24.212ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
602024-05-15T03:45:24.212ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
612024-05-15T03:45:24.212ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
622024-05-15T03:45:24.212ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
632024-05-15T03:45:24.213ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
642024-05-15T03:45:24.213ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
652024-05-15T03:45:24.214ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3701, tv_nsec: 642021896 } })
662024-05-15T03:45:24.214ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:92
672024-05-15T03:45:24.214ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.850329ms iteration = 8
682024-05-15T03:45:24.216ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3701, tv_nsec: 644622402 } })
692024-05-15T03:45:24.216ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1022
702024-05-15T03:45:24.217ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
712024-05-15T03:45:24.217ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
722024-05-15T03:45:24.217ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
732024-05-15T03:45:24.218ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
742024-05-15T03:45:24.218ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
752024-05-15T03:45:24.218ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
762024-05-15T03:45:24.218ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
772024-05-15T03:45:24.218ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
782024-05-15T03:45:24.218ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
792024-05-15T03:45:24.218ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
802024-05-15T03:45:24.218ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
812024-05-15T03:45:24.219ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
822024-05-15T03:45:24.219ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
832024-05-15T03:45:24.219ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1022
842024-05-15T03:45:24.219ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
852024-05-15T03:45:24.219ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
862024-05-15T03:45:24.219ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
872024-05-15T03:45:24.219ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
882024-05-15T03:45:24.220ZWARNnexus (BackgroundTasks): physical disk adoption task started background_task = physical_disk_adoption file = nexus/src/app/background/physical_disk_adoption.rs:100
892024-05-15T03:45:24.220ZWARNnexus (BackgroundTasks): Physical Disk Adoption: skipped background_task = physical_disk_adoption file = nexus/src/app/background/physical_disk_adoption.rs:104 reason = no inventory
902024-05-15T03:45:24.220ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 9.916241ms iteration = 8
912024-05-15T03:45:24.220ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
922024-05-15T03:45:24.220ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3701, tv_nsec: 648735234 } })
932024-05-15T03:45:24.221ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
942024-05-15T03:45:24.222ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1022 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4
952024-05-15T03:45:24.222ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
962024-05-15T03:45:24.224ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0b55a4be-1fdc-4b91-a3a7-f964eab4522f, lookup_type: ById(0b55a4be-1fdc-4b91-a3a7-f964eab4522f) } result = Ok(())
972024-05-15T03:45:24.224ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
982024-05-15T03:45:24.225ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
992024-05-15T03:45:24.231ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f original_time_created = 2024-05-15 03:41:43.607350 UTC target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f time_created = 2024-05-15 03:41:43.607350 UTC
1002024-05-15T03:45:24.231ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.946163ms iteration = 22
1012024-05-15T03:45:24.243ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1022024-05-15T03:45:24.243ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1032024-05-15T03:45:24.244ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1042024-05-15T03:45:24.245ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1052024-05-15T03:45:24.246ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1062024-05-15T03:45:24.249ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 10925274-3dbb-4558-9fc7-9ec149150c20, lookup_type: ById(10925274-3dbb-4558-9fc7-9ec149150c20) } result = Ok(())
1072024-05-15T03:45:24.249ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
1082024-05-15T03:45:24.249ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
1092024-05-15T03:45:24.250ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 39.704855ms iteration = 8
1102024-05-15T03:45:24.284ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/instance_watcher.rs:469 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0
1112024-05-15T03:45:24.284ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 72.943664ms iteration = 8
1122024-05-15T03:45:24.297ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1030 zone_address = fd00:1122:3344:101::2
1132024-05-15T03:45:24.297ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
1142024-05-15T03:45:24.298ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1030 zone_address = fd00:1122:3344:101::2
1152024-05-15T03:45:24.298ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
1162024-05-15T03:45:24.298ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "17c964b8-5c41-4157-8f96-55879d2dc8fc", "content-length": "26", "date": "Wed, 15 May 2024 03:45:24 GMT"} })
1172024-05-15T03:45:24.298ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1035 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
1182024-05-15T03:45:24.298ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "9d6e107e-1b18-4c1e-a89f-bdad3880ab32", "content-length": "26", "date": "Wed, 15 May 2024 03:45:24 GMT"} })
1192024-05-15T03:45:24.298ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1035 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
1202024-05-15T03:45:24.300ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1067 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
1212024-05-15T03:45:24.301ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1067 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
1222024-05-15T03:45:24.301ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1030 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 zone_address = fd00:1122:3344:101::2
1232024-05-15T03:45:24.301ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
1242024-05-15T03:45:24.302ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "3e6eaa55-31a3-4b36-86f0-b1a5e5bf6244", "content-length": "26", "date": "Wed, 15 May 2024 03:45:24 GMT"} })
1252024-05-15T03:45:24.302ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1035 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
1262024-05-15T03:45:24.304ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1067 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4
1272024-05-15T03:45:24.392ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
1282024-05-15T03:45:24.392ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "958b240c-4809-47b1-a1c1-15984fbf874c", "content-length": "2", "date": "Wed, 15 May 2024 03:45:24 GMT"} })
1292024-05-15T03:45:24.393ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
1302024-05-15T03:45:24.393ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
1312024-05-15T03:45:24.394ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 184.188749ms iteration = 8
1322024-05-15T03:45:24.394ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 184.573933ms iteration = 8
1332024-05-15T03:45:24.454ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
1342024-05-15T03:45:24.455ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
1352024-05-15T03:45:24.456ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:144 port = SwitchPort { id: 89eb00e0-b12a-4307-aac5-625fd364b560, rack_id: 76c05259-5403-45cc-a1c6-20f84d7e54b4, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(7f8c724a-b675-4fd5-8a78-6b7ffc7a3247) } rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0
1362024-05-15T03:45:24.456ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
1372024-05-15T03:45:24.456ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
1382024-05-15T03:45:24.463ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
1392024-05-15T03:45:24.463ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/port/qsfp0/settings", query: Some("tag=omicron"), fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "944ba184-365a-4b03-80eb-cf80e3e78cfd", "content-length": "121", "date": "Wed, 15 May 2024 03:45:24 GMT"} })
1402024-05-15T03:45:24.463ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1643 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: None, kr: false, lane: Some(LinkId(0)), speed: Speed40G } }} } switch_port_id = "qsfp0"
1412024-05-15T03:45:24.463ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1670 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 7f8c724a-b675-4fd5-8a78-6b7ffc7a3247
1422024-05-15T03:45:24.464ZERROnexus (BackgroundTasks): unable to retrieve routes from switch background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1782 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0
1432024-05-15T03:45:24.464ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:369 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 routes = {}
1442024-05-15T03:45:24.464ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:373 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None)}}
1452024-05-15T03:45:24.464ZWARNnexus (BackgroundTasks): no discovered routes from switch. it is possible that an earlier api call failed. background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1511 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0
1462024-05-15T03:45:24.464ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:382 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 routes = {}
1472024-05-15T03:45:24.464ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:388 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 routes = {}
1482024-05-15T03:45:24.464ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:408 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4
1492024-05-15T03:45:24.464ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
1502024-05-15T03:45:24.464ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
1512024-05-15T03:45:24.466ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
1522024-05-15T03:45:24.466ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv4", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "83641934-eb0c-4707-ada9-e52f468f60e9", "content-length": "2", "date": "Wed, 15 May 2024 03:45:24 GMT"} })
1532024-05-15T03:45:24.466ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
1542024-05-15T03:45:24.466ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv6", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "58080a1a-29ad-4db8-8526-8689839dc5d4", "content-length": "52", "date": "Wed, 15 May 2024 03:45:24 GMT"} })
1552024-05-15T03:45:24.466ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [V4(Ipv4Network { addr: 10.151.1.174, prefix: 24 })], port: "qsfp0" }] file = nexus/src/app/background/sync_switch_configuration.rs:457 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0
1562024-05-15T03:45:24.466ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
1572024-05-15T03:45:24.529ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/switch-ports", query: None, fragment: None }, status: 204, headers: {"x-request-id": "100b2214-346d-43e6-b5cc-38d4b77a366d", "date": "Wed, 15 May 2024 03:45:24 GMT"} })
1582024-05-15T03:45:24.529ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:759 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch = Switch0
1592024-05-15T03:45:24.529ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
1602024-05-15T03:45:24.530ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/network-bootstore-config", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "cde65539-735d-4bbe-94be-f669bbbe5f3c", "content-length": "467", "date": "Wed, 15 May 2024 03:45:24 GMT"} })
1612024-05-15T03:45:24.530ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
1622024-05-15T03:45:24.530ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
1632024-05-15T03:45:24.533ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
1642024-05-15T03:45:24.533ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
1652024-05-15T03:45:24.536ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
1662024-05-15T03:45:24.536ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
1672024-05-15T03:45:24.537ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
1682024-05-15T03:45:24.537ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
1692024-05-15T03:45:24.544ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 333.967277ms iteration = 8
1702024-05-15T03:45:24.704ZDEBGnexus (producer-registration-task): registering / renewing oximeter producer lease with Nexus
1712024-05-15T03:45:24.704ZDEBGnexus (producer-registration-task): using nexus address for registration addr = [fd00:1122:3344:101::c]:12221
1722024-05-15T03:45:24.735ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::c]:12221/metrics/producers
1732024-05-15T03:45:24.735ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::c]:12221 remote_addr = [fd00:1122:3344:101::c]:63711
1742024-05-15T03:45:24.735ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::c]:63711 req_id = 37899086-d4d1-406b-b41f-77a00f2a94d1 roles = RoleSet { roles: {} } uri = /metrics/producers
1752024-05-15T03:45:24.735ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 235 local_addr = [fd00:1122:3344:101::c]:55127 method = GET remote_addr = [fd00:1122:3344:101::d]:37120 req_id = 7a1e897a-22ad-40d3-aa61-f84b34d50ed6 response_code = 200 uri = //86c56a1d-af69-43b6-8206-e5d1baefadb2
1762024-05-15T03:45:24.735ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::c]:63711 req_id = 37899086-d4d1-406b-b41f-77a00f2a94d1 resource = Database result = Ok(()) uri = /metrics/producers
1772024-05-15T03:45:24.767ZINFOnexus: registered oximeter collector client file = nexus/src/app/oximeter.rs:338 id = f12aa7e2-1873-452c-b215-b2593f228861
1782024-05-15T03:45:24.768ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::c]:63711 req_id = 37899086-d4d1-406b-b41f-77a00f2a94d1 roles = RoleSet { roles: {} } uri = /metrics/producers
1792024-05-15T03:45:24.768ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::c]:63711 req_id = 37899086-d4d1-406b-b41f-77a00f2a94d1 resource = Database result = Ok(()) uri = /metrics/producers
1802024-05-15T03:45:24.803ZDEBGnexus: client request body = Some(Body) method = POST oximeter-collector = f12aa7e2-1873-452c-b215-b2593f228861 uri = http://[fd00:1122:3344:101::d]:12223/producers
1812024-05-15T03:45:24.804ZDEBGnexus: client response oximeter-collector = f12aa7e2-1873-452c-b215-b2593f228861 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::d)), port: Some(12223), path: "/producers", query: None, fragment: None }, status: 204, headers: {"x-request-id": "5ed58887-ca9e-43d3-8072-8760c6068056", "date": "Wed, 15 May 2024 03:45:24 GMT"} })
1822024-05-15T03:45:24.804ZINFOnexus: assigned collector to new producer collector_id = f12aa7e2-1873-452c-b215-b2593f228861 file = nexus/src/app/oximeter.rs:125 producer_id = 86c56a1d-af69-43b6-8206-e5d1baefadb2
1832024-05-15T03:45:24.805ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 70353 local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::c]:63711 req_id = 37899086-d4d1-406b-b41f-77a00f2a94d1 response_code = 201 uri = /metrics/producers
1842024-05-15T03:45:24.805ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::c)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, status: 201, headers: {"content-type": "application/json", "x-request-id": "37899086-d4d1-406b-b41f-77a00f2a94d1", "content-length": "41", "date": "Wed, 15 May 2024 03:45:24 GMT"} })
1852024-05-15T03:45:24.807ZDEBGnexus (producer-registration-task): registered with nexus successfully
1862024-05-15T03:45:24.807ZDEBGnexus (producer-registration-task): pausing until time to renew lease lease_duration = 600s wait_period = 150s
1872024-05-15T03:45:34.210ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 23 reason = Timeout
1882024-05-15T03:45:34.210ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
1892024-05-15T03:45:34.210ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
1902024-05-15T03:45:34.212ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1912024-05-15T03:45:34.214ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
1922024-05-15T03:45:34.214ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
1932024-05-15T03:45:34.215ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
1942024-05-15T03:45:34.216ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
1952024-05-15T03:45:34.216ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
1962024-05-15T03:45:34.217ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
1972024-05-15T03:45:34.220ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0b55a4be-1fdc-4b91-a3a7-f964eab4522f, lookup_type: ById(0b55a4be-1fdc-4b91-a3a7-f964eab4522f) } result = Ok(())
1982024-05-15T03:45:34.220ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
1992024-05-15T03:45:34.220ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2002024-05-15T03:45:34.226ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f original_time_created = 2024-05-15 03:41:43.607350 UTC target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f time_created = 2024-05-15 03:41:43.607350 UTC
2012024-05-15T03:45:34.226ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 16.083845ms iteration = 23
2022024-05-15T03:45:34.805ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 141 local_addr = [fd00:1122:3344:101::c]:55127 method = GET remote_addr = [fd00:1122:3344:101::d]:37120 req_id = c17f7ffc-4d02-4d52-9a88-d8cf64a85919 response_code = 200 uri = //86c56a1d-af69-43b6-8206-e5d1baefadb2
2032024-05-15T03:45:44.210ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 24 reason = Timeout
2042024-05-15T03:45:44.210ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2052024-05-15T03:45:44.210ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2062024-05-15T03:45:44.212ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2072024-05-15T03:45:44.215ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
2082024-05-15T03:45:44.215ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2092024-05-15T03:45:44.215ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2102024-05-15T03:45:44.216ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2112024-05-15T03:45:44.217ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2122024-05-15T03:45:44.218ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2132024-05-15T03:45:44.220ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0b55a4be-1fdc-4b91-a3a7-f964eab4522f, lookup_type: ById(0b55a4be-1fdc-4b91-a3a7-f964eab4522f) } result = Ok(())
2142024-05-15T03:45:44.220ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2152024-05-15T03:45:44.220ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2162024-05-15T03:45:44.227ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f original_time_created = 2024-05-15 03:41:43.607350 UTC target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f time_created = 2024-05-15 03:41:43.607350 UTC
2172024-05-15T03:45:44.227ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 16.476858ms iteration = 24
2182024-05-15T03:45:44.806ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 168 local_addr = [fd00:1122:3344:101::c]:55127 method = GET remote_addr = [fd00:1122:3344:101::d]:37120 req_id = 00ef5d28-e5b7-4fcd-883f-d2f7dd8141ce response_code = 200 uri = //86c56a1d-af69-43b6-8206-e5d1baefadb2
2192024-05-15T03:45:54.210ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 25 reason = Timeout
2202024-05-15T03:45:54.210ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 5 reason = Timeout
2212024-05-15T03:45:54.210ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
2222024-05-15T03:45:54.210ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2232024-05-15T03:45:54.210ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
2242024-05-15T03:45:54.210ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 5 reason = Timeout
2252024-05-15T03:45:54.210ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 6 reason = Timeout
2262024-05-15T03:45:54.210ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/blueprint_execution.rs:65 reason = blueprint disabled target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f
2272024-05-15T03:45:54.210ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 17.51\u{b5}s iteration = 6
2282024-05-15T03:45:54.210ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 5 reason = Timeout
2292024-05-15T03:45:54.210ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 7 reason = Timeout
2302024-05-15T03:45:54.210ZWARNnexus (BackgroundTasks): Metric producer GC: statically disabled pending omicron#5284 background_task = metrics_producer_gc file = nexus/src/app/background/metrics_producer_gc.rs:43
2312024-05-15T03:45:54.210ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 8.02\u{b5}s iteration = 5
2322024-05-15T03:45:54.210ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 9 reason = Timeout
2332024-05-15T03:45:54.210ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 5 reason = Timeout
2342024-05-15T03:45:54.210ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
2352024-05-15T03:45:54.210ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 5 reason = Timeout
2362024-05-15T03:45:54.210ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
2372024-05-15T03:45:54.210ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 5 reason = Timeout
2382024-05-15T03:45:54.210ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
2392024-05-15T03:45:54.210ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
2402024-05-15T03:45:54.210ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 7 reason = Timeout
2412024-05-15T03:45:54.210ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 9 reason = Timeout
2422024-05-15T03:45:54.210ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:46
2432024-05-15T03:45:54.210ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 9 reason = Timeout
2442024-05-15T03:45:54.210ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
2452024-05-15T03:45:54.211ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
2462024-05-15T03:45:54.211ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
2472024-05-15T03:45:54.211ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
2482024-05-15T03:45:54.211ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2492024-05-15T03:45:54.211ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 9 reason = Timeout
2502024-05-15T03:45:54.211ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
2512024-05-15T03:45:54.211ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 9 reason = Timeout
2522024-05-15T03:45:54.211ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 9 reason = Timeout
2532024-05-15T03:45:54.211ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:44
2542024-05-15T03:45:54.211ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:48
2552024-05-15T03:45:54.211ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 9 reason = Timeout
2562024-05-15T03:45:54.211ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
2572024-05-15T03:45:54.211ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 8.17\u{b5}s iteration = 9
2582024-05-15T03:45:54.211ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
2592024-05-15T03:45:54.211ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
2602024-05-15T03:45:54.211ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
2612024-05-15T03:45:54.211ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("3a1867b9-e5ea-4194-bfc7-0db4fb07d69e.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("9321e01e-813d-4e4f-8668-60649e9711dc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("9321e01e-813d-4e4f-8668-60649e9711dc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::8)) }], valid_until: Instant { tv_sec: 3731, tv_nsec: 638928854 } })
2622024-05-15T03:45:54.211ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
2632024-05-15T03:45:54.211ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
2642024-05-15T03:45:54.211ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
2652024-05-15T03:45:54.212ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 9 reason = Timeout
2662024-05-15T03:45:54.212ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
2672024-05-15T03:45:54.213ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2682024-05-15T03:45:54.213ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
2692024-05-15T03:45:54.213ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("2b87d77b-ee3f-43c8-bced-dd105bbcdaa3.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("510fd65b-f60d-4faa-85be-82d26f4dd427.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("65e0e735-7f1e-479a-8127-a80c8289fcdb.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("65e0e735-7f1e-479a-8127-a80c8289fcdb.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:1::1)) }], valid_until: Instant { tv_sec: 3731, tv_nsec: 640467170 } })
2702024-05-15T03:45:54.213ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
2712024-05-15T03:45:54.213ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2722024-05-15T03:45:54.213ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
2732024-05-15T03:45:54.213ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2742024-05-15T03:45:54.214ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2752024-05-15T03:45:54.214ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
2762024-05-15T03:45:54.214ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2772024-05-15T03:45:54.215ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
2782024-05-15T03:45:54.215ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:92
2792024-05-15T03:45:54.215ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.757898ms iteration = 9
2802024-05-15T03:45:54.215ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2812024-05-15T03:45:54.216ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2822024-05-15T03:45:54.216ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
2832024-05-15T03:45:54.217ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
2842024-05-15T03:45:54.217ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 7.037322ms iteration = 5
2852024-05-15T03:45:54.219ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3731, tv_nsec: 646381230 } })
2862024-05-15T03:45:54.219ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
2872024-05-15T03:45:54.219ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
2882024-05-15T03:45:54.219ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
2892024-05-15T03:45:54.219ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
2902024-05-15T03:45:54.219ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
2912024-05-15T03:45:54.220ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
2922024-05-15T03:45:54.220ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
2932024-05-15T03:45:54.220ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
2942024-05-15T03:45:54.220ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
2952024-05-15T03:45:54.220ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
2962024-05-15T03:45:54.221ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
2972024-05-15T03:45:54.221ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
2982024-05-15T03:45:54.221ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
2992024-05-15T03:45:54.221ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
3002024-05-15T03:45:54.221ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
3012024-05-15T03:45:54.221ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
3022024-05-15T03:45:54.221ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 11.050542ms iteration = 5
3032024-05-15T03:45:54.222ZWARNnexus (BackgroundTasks): physical disk adoption task started background_task = physical_disk_adoption file = nexus/src/app/background/physical_disk_adoption.rs:100
3042024-05-15T03:45:54.222ZWARNnexus (BackgroundTasks): Physical Disk Adoption: skipped background_task = physical_disk_adoption file = nexus/src/app/background/physical_disk_adoption.rs:104 reason = no inventory
3052024-05-15T03:45:54.222ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 11.215154ms iteration = 9
3062024-05-15T03:45:54.222ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3072024-05-15T03:45:54.222ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
3082024-05-15T03:45:54.223ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3092024-05-15T03:45:54.223ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
3102024-05-15T03:45:54.223ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
3112024-05-15T03:45:54.223ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
3122024-05-15T03:45:54.223ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
3132024-05-15T03:45:54.223ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
3142024-05-15T03:45:54.223ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
3152024-05-15T03:45:54.223ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
3162024-05-15T03:45:54.224ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
3172024-05-15T03:45:54.224ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
3182024-05-15T03:45:54.224ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
3192024-05-15T03:45:54.224ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
3202024-05-15T03:45:54.224ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3212024-05-15T03:45:54.225ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3731, tv_nsec: 652359691 } })
3222024-05-15T03:45:54.225ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3232024-05-15T03:45:54.225ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3242024-05-15T03:45:54.225ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
3252024-05-15T03:45:54.225ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
3262024-05-15T03:45:54.225ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
3272024-05-15T03:45:54.225ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
3282024-05-15T03:45:54.225ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
3292024-05-15T03:45:54.225ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
3302024-05-15T03:45:54.226ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
3312024-05-15T03:45:54.226ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1022
3322024-05-15T03:45:54.226ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/instance_watcher.rs:469 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0
3332024-05-15T03:45:54.226ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 13.830301ms iteration = 9
3342024-05-15T03:45:54.227ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3352024-05-15T03:45:54.229ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0b55a4be-1fdc-4b91-a3a7-f964eab4522f, lookup_type: ById(0b55a4be-1fdc-4b91-a3a7-f964eab4522f) } result = Ok(())
3362024-05-15T03:45:54.229ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
3372024-05-15T03:45:54.230ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
3382024-05-15T03:45:54.231ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3731, tv_nsec: 658388783 } })
3392024-05-15T03:45:54.231ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
3402024-05-15T03:45:54.231ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
3412024-05-15T03:45:54.231ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
3422024-05-15T03:45:54.231ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
3432024-05-15T03:45:54.231ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
3442024-05-15T03:45:54.231ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
3452024-05-15T03:45:54.232ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
3462024-05-15T03:45:54.232ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
3472024-05-15T03:45:54.232ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1022
3482024-05-15T03:45:54.232ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
3492024-05-15T03:45:54.233ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
3502024-05-15T03:45:54.233ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 3698e770-e01a-43e5-856e-b62ced4fb671 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
3512024-05-15T03:45:54.233ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
3522024-05-15T03:45:54.233ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
3532024-05-15T03:45:54.234ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1022 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4
3542024-05-15T03:45:54.234ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
3552024-05-15T03:45:54.234ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 52cc15d1-4ce1-4e38-8c6c-f9f099a8a5be dns_zone_name = oxide.test found_so_far = 0 version = 2
3562024-05-15T03:45:54.234ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
3572024-05-15T03:45:54.234ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
3582024-05-15T03:45:54.235ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3592024-05-15T03:45:54.236ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3602024-05-15T03:45:54.239ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
3612024-05-15T03:45:54.239ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
3622024-05-15T03:45:54.239ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
3632024-05-15T03:45:54.239ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
3642024-05-15T03:45:54.240ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
3652024-05-15T03:45:54.240ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
3662024-05-15T03:45:54.245ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f original_time_created = 2024-05-15 03:41:43.607350 UTC target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f time_created = 2024-05-15 03:41:43.607350 UTC
3672024-05-15T03:45:54.245ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 34.865736ms iteration = 25
3682024-05-15T03:45:54.249ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 3698e770-e01a-43e5-856e-b62ced4fb671 dns_zone_name = control-plane.oxide.internal found_so_far = 49 version = 1
3692024-05-15T03:45:54.249ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
3702024-05-15T03:45:54.249ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2024-05-15 03:41:56.187733 UTC dns_group = internal generation = 1
3712024-05-15T03:45:54.249ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 38.400182ms iteration = 5
3722024-05-15T03:45:54.249ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 52cc15d1-4ce1-4e38-8c6c-f9f099a8a5be dns_zone_name = oxide.test found_so_far = 1 version = 2
3732024-05-15T03:45:54.249ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
3742024-05-15T03:45:54.249ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2024-05-15 03:41:57.099854 UTC dns_group = external generation = 2
3752024-05-15T03:45:54.249ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 38.555083ms iteration = 5
3762024-05-15T03:45:54.255ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
3772024-05-15T03:45:54.255ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
3782024-05-15T03:45:54.257ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
3792024-05-15T03:45:54.257ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
3802024-05-15T03:45:54.258ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3812024-05-15T03:45:54.263ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 10925274-3dbb-4558-9fc7-9ec149150c20, lookup_type: ById(10925274-3dbb-4558-9fc7-9ec149150c20) } result = Ok(())
3822024-05-15T03:45:54.263ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
3832024-05-15T03:45:54.263ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
3842024-05-15T03:45:54.265ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 53.544696ms iteration = 9
3852024-05-15T03:45:54.274ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
3862024-05-15T03:45:54.274ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
3872024-05-15T03:45:54.275ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
3882024-05-15T03:45:54.278ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
3892024-05-15T03:45:54.278ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
3902024-05-15T03:45:54.278ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
3912024-05-15T03:45:54.331ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2024-05-15T03:41:54.396604Z, time_modified: 2024-05-15T03:41:54.396604Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 39f7e4ba-533e-4258-9f79-1873f513f101, db_silo: Silo { identity: SiloIdentity { id: 39f7e4ba-533e-4258-9f79-1873f513f101, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-05-15T03:41:57.052187Z, time_modified: 2024-05-15T03:41:57.052187Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "370e7f12febb2dc0abbebc4192727b23c734c770fdab3368ed3fd759491f6fed" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 39f7e4ba-533e-4258-9f79-1873f513f101, db_silo: Silo { identity: SiloIdentity { id: 39f7e4ba-533e-4258-9f79-1873f513f101, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-05-15T03:41:57.052187Z, time_modified: 2024-05-15T03:41:57.052187Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "370e7f12febb2dc0abbebc4192727b23c734c770fdab3368ed3fd759491f6fed" }] }) }
3922024-05-15T03:45:54.332ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 121.312447ms iteration = 5
3932024-05-15T03:45:54.395ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1030 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 zone_address = fd00:1122:3344:101::2
3942024-05-15T03:45:54.395ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
3952024-05-15T03:45:54.395ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "be286c66-788b-4181-b20b-cd062c40028f", "content-length": "26", "date": "Wed, 15 May 2024 03:45:54 GMT"} })
3962024-05-15T03:45:54.395ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1035 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
3972024-05-15T03:45:54.397ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1067 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4
3982024-05-15T03:45:54.458ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
3992024-05-15T03:45:54.473ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
4002024-05-15T03:45:54.513ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1030 zone_address = fd00:1122:3344:101::2
4012024-05-15T03:45:54.513ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
4022024-05-15T03:45:54.513ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "c2e112d1-1cd9-480b-825f-a3fb318da2d7", "content-length": "26", "date": "Wed, 15 May 2024 03:45:54 GMT"} })
4032024-05-15T03:45:54.513ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1035 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
4042024-05-15T03:45:54.514ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1067 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
4052024-05-15T03:45:54.537ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1030 zone_address = fd00:1122:3344:101::2
4062024-05-15T03:45:54.537ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
4072024-05-15T03:45:54.537ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "5849ca8a-7deb-4df2-9454-15e07c8c3bcd", "content-length": "26", "date": "Wed, 15 May 2024 03:45:54 GMT"} })
4082024-05-15T03:45:54.537ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1035 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
4092024-05-15T03:45:54.539ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1067 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
4102024-05-15T03:45:54.630ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
4112024-05-15T03:45:54.631ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::8)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "29206485-cb3c-48f6-b1b6-220dcf808f40", "date": "Wed, 15 May 2024 03:45:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
4122024-05-15T03:45:54.631ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
4132024-05-15T03:45:54.632ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::9)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "2e95b9ff-36f4-4c7d-9455-fcb4d996fa5b", "date": "Wed, 15 May 2024 03:45:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
4142024-05-15T03:45:54.632ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
4152024-05-15T03:45:54.633ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
4162024-05-15T03:45:54.633ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 422.968855ms iteration = 7
4172024-05-15T03:45:54.678ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
4182024-05-15T03:45:54.678ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "5bd7ba8d-6cb4-46b7-b7e4-030c7cbfb8bb", "content-length": "2", "date": "Wed, 15 May 2024 03:45:54 GMT"} })
4192024-05-15T03:45:54.680ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
4202024-05-15T03:45:54.681ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
4212024-05-15T03:45:54.682ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 471.431339ms iteration = 9
4222024-05-15T03:45:54.715ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
4232024-05-15T03:45:54.762ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
4242024-05-15T03:45:54.762ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
4252024-05-15T03:45:54.764ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:144 port = SwitchPort { id: 89eb00e0-b12a-4307-aac5-625fd364b560, rack_id: 76c05259-5403-45cc-a1c6-20f84d7e54b4, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(7f8c724a-b675-4fd5-8a78-6b7ffc7a3247) } rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0
4262024-05-15T03:45:54.764ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
4272024-05-15T03:45:54.764ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
4282024-05-15T03:45:54.772ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
4292024-05-15T03:45:54.773ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/port/qsfp0/settings", query: Some("tag=omicron"), fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "297597cc-f011-4bb1-a53c-6f9a3b5e05d8", "content-length": "121", "date": "Wed, 15 May 2024 03:45:54 GMT"} })
4302024-05-15T03:45:54.773ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1643 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: None, kr: false, lane: Some(LinkId(0)), speed: Speed40G } }} } switch_port_id = "qsfp0"
4312024-05-15T03:45:54.773ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1670 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 7f8c724a-b675-4fd5-8a78-6b7ffc7a3247
4322024-05-15T03:45:54.773ZERROnexus (BackgroundTasks): unable to retrieve routes from switch background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1782 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0
4332024-05-15T03:45:54.773ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:369 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 routes = {}
4342024-05-15T03:45:54.773ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:373 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None)}}
4352024-05-15T03:45:54.773ZWARNnexus (BackgroundTasks): no discovered routes from switch. it is possible that an earlier api call failed. background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1511 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0
4362024-05-15T03:45:54.773ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:382 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 routes = {}
4372024-05-15T03:45:54.773ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:388 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 routes = {}
4382024-05-15T03:45:54.773ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:408 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4
4392024-05-15T03:45:54.773ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
4402024-05-15T03:45:54.773ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
4412024-05-15T03:45:54.774ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 564.176386ms iteration = 9
4422024-05-15T03:45:54.775ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
4432024-05-15T03:45:54.776ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv4", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "975ea568-b50d-4b5e-b5fa-8f4ce6aa168f", "content-length": "2", "date": "Wed, 15 May 2024 03:45:54 GMT"} })
4442024-05-15T03:45:54.776ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
4452024-05-15T03:45:54.776ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv6", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "8aa1f4e2-4a30-4931-ab98-ae6a98611378", "content-length": "52", "date": "Wed, 15 May 2024 03:45:54 GMT"} })
4462024-05-15T03:45:54.776ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [V4(Ipv4Network { addr: 10.151.1.174, prefix: 24 })], port: "qsfp0" }] file = nexus/src/app/background/sync_switch_configuration.rs:457 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0
4472024-05-15T03:45:54.776ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
4482024-05-15T03:45:54.793ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
4492024-05-15T03:45:54.794ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:1::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "ee4bcf44-438f-4dbb-8420-660c8402e01f", "date": "Wed, 15 May 2024 03:45:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
4502024-05-15T03:45:54.794ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
4512024-05-15T03:45:54.795ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:3::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "81a2807d-e733-4d4b-ac9b-153a52a1213b", "date": "Wed, 15 May 2024 03:45:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
4522024-05-15T03:45:54.795ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
4532024-05-15T03:45:54.796ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:2::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "f613fdc4-c0c8-44f8-b0c5-f7cb70d29a7f", "date": "Wed, 15 May 2024 03:45:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
4542024-05-15T03:45:54.796ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
4552024-05-15T03:45:54.797ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
4562024-05-15T03:45:54.797ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 587.12082ms iteration = 7
4572024-05-15T03:45:54.805ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 132 local_addr = [fd00:1122:3344:101::c]:55127 method = GET remote_addr = [fd00:1122:3344:101::d]:37120 req_id = 4d81217f-5b25-49be-8441-960dcb6ed777 response_code = 200 uri = //86c56a1d-af69-43b6-8206-e5d1baefadb2
4582024-05-15T03:45:54.872ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/switch-ports", query: None, fragment: None }, status: 204, headers: {"x-request-id": "4ea0f2ae-918f-4d65-96b1-b2fd3443e0a5", "date": "Wed, 15 May 2024 03:45:54 GMT"} })
4592024-05-15T03:45:54.872ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:759 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch = Switch0
4602024-05-15T03:45:54.872ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
4612024-05-15T03:45:54.872ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/network-bootstore-config", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "5be7f551-0193-4928-aba2-9c90d6edc435", "content-length": "467", "date": "Wed, 15 May 2024 03:45:54 GMT"} })
4622024-05-15T03:45:54.872ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
4632024-05-15T03:45:54.873ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
4642024-05-15T03:45:54.876ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
4652024-05-15T03:45:54.876ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
4662024-05-15T03:45:54.879ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
4672024-05-15T03:45:54.879ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
4682024-05-15T03:45:54.881ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
4692024-05-15T03:45:54.881ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
4702024-05-15T03:45:54.888ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 676.610503ms iteration = 9
4712024-05-15T03:46:04.211ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 26 reason = Timeout
4722024-05-15T03:46:04.211ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4732024-05-15T03:46:04.211ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4742024-05-15T03:46:04.213ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4752024-05-15T03:46:04.215ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
4762024-05-15T03:46:04.215ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4772024-05-15T03:46:04.215ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4782024-05-15T03:46:04.217ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4792024-05-15T03:46:04.217ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4802024-05-15T03:46:04.218ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4812024-05-15T03:46:04.221ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0b55a4be-1fdc-4b91-a3a7-f964eab4522f, lookup_type: ById(0b55a4be-1fdc-4b91-a3a7-f964eab4522f) } result = Ok(())
4822024-05-15T03:46:04.221ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4832024-05-15T03:46:04.221ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4842024-05-15T03:46:04.237ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f original_time_created = 2024-05-15 03:41:43.607350 UTC target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f time_created = 2024-05-15 03:41:43.607350 UTC
4852024-05-15T03:46:04.237ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 25.776403ms iteration = 26
4862024-05-15T03:46:04.805ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 163 local_addr = [fd00:1122:3344:101::c]:55127 method = GET remote_addr = [fd00:1122:3344:101::d]:37120 req_id = 795464ab-3a1d-4c2a-b6d6-3270955c135b response_code = 200 uri = //86c56a1d-af69-43b6-8206-e5d1baefadb2
4872024-05-15T03:46:14.211ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 27 reason = Timeout
4882024-05-15T03:46:14.211ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4892024-05-15T03:46:14.211ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4902024-05-15T03:46:14.213ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4912024-05-15T03:46:14.215ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
4922024-05-15T03:46:14.215ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4932024-05-15T03:46:14.215ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4942024-05-15T03:46:14.217ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4952024-05-15T03:46:14.217ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
4962024-05-15T03:46:14.218ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
4972024-05-15T03:46:14.221ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0b55a4be-1fdc-4b91-a3a7-f964eab4522f, lookup_type: ById(0b55a4be-1fdc-4b91-a3a7-f964eab4522f) } result = Ok(())
4982024-05-15T03:46:14.221ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
4992024-05-15T03:46:14.221ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
5002024-05-15T03:46:14.229ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f original_time_created = 2024-05-15 03:41:43.607350 UTC target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f time_created = 2024-05-15 03:41:43.607350 UTC
5012024-05-15T03:46:14.229ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 17.761751ms iteration = 27
5022024-05-15T03:46:14.806ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 690 local_addr = [fd00:1122:3344:101::c]:55127 method = GET remote_addr = [fd00:1122:3344:101::d]:37120 req_id = 770e9d4e-aed5-4745-bb52-8bb0d07ab082 response_code = 200 uri = //86c56a1d-af69-43b6-8206-e5d1baefadb2
5032024-05-15T03:46:24.211ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 28 reason = Timeout
5042024-05-15T03:46:24.211ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 10 reason = Timeout
5052024-05-15T03:46:24.211ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
5062024-05-15T03:46:24.211ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 10 reason = Timeout
5072024-05-15T03:46:24.211ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
5082024-05-15T03:46:24.211ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
5092024-05-15T03:46:24.211ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 10 reason = Timeout
5102024-05-15T03:46:24.211ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:46
5112024-05-15T03:46:24.211ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
5122024-05-15T03:46:24.211ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
5132024-05-15T03:46:24.211ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
5142024-05-15T03:46:24.212ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 10 reason = Timeout
5152024-05-15T03:46:24.212ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 10 reason = Timeout
5162024-05-15T03:46:24.212ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 10 reason = Timeout
5172024-05-15T03:46:24.212ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
5182024-05-15T03:46:24.212ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
5192024-05-15T03:46:24.212ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5202024-05-15T03:46:24.212ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 10 reason = Timeout
5212024-05-15T03:46:24.212ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:44
5222024-05-15T03:46:24.212ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:48
5232024-05-15T03:46:24.212ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 11.99\u{b5}s iteration = 10
5242024-05-15T03:46:24.212ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
5252024-05-15T03:46:24.212ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5262024-05-15T03:46:24.212ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
5272024-05-15T03:46:24.212ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 10 reason = Timeout
5282024-05-15T03:46:24.212ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
5292024-05-15T03:46:24.213ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
5302024-05-15T03:46:24.213ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5312024-05-15T03:46:24.213ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3761, tv_nsec: 640172397 } })
5322024-05-15T03:46:24.213ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3761, tv_nsec: 640441129 } })
5332024-05-15T03:46:24.214ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5342024-05-15T03:46:24.214ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5352024-05-15T03:46:24.214ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5362024-05-15T03:46:24.214ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5372024-05-15T03:46:24.215ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1022
5382024-05-15T03:46:24.215ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:92
5392024-05-15T03:46:24.215ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.082072ms iteration = 10
5402024-05-15T03:46:24.216ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1022
5412024-05-15T03:46:24.217ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
5422024-05-15T03:46:24.217ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
5432024-05-15T03:46:24.218ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
5442024-05-15T03:46:24.218ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
5452024-05-15T03:46:24.218ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
5462024-05-15T03:46:24.218ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
5472024-05-15T03:46:24.219ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
5482024-05-15T03:46:24.219ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
5492024-05-15T03:46:24.219ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
5502024-05-15T03:46:24.220ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
5512024-05-15T03:46:24.220ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
5522024-05-15T03:46:24.220ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
5532024-05-15T03:46:24.220ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
5542024-05-15T03:46:24.220ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
5552024-05-15T03:46:24.221ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
5562024-05-15T03:46:24.221ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5572024-05-15T03:46:24.221ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5582024-05-15T03:46:24.221ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5592024-05-15T03:46:24.222ZWARNnexus (BackgroundTasks): physical disk adoption task started background_task = physical_disk_adoption file = nexus/src/app/background/physical_disk_adoption.rs:100
5602024-05-15T03:46:24.222ZWARNnexus (BackgroundTasks): Physical Disk Adoption: skipped background_task = physical_disk_adoption file = nexus/src/app/background/physical_disk_adoption.rs:104 reason = no inventory
5612024-05-15T03:46:24.222ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 9.716919ms iteration = 10
5622024-05-15T03:46:24.222ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3761, tv_nsec: 649492552 } })
5632024-05-15T03:46:24.222ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5642024-05-15T03:46:24.223ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5652024-05-15T03:46:24.223ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1022 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4
5662024-05-15T03:46:24.225ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0b55a4be-1fdc-4b91-a3a7-f964eab4522f, lookup_type: ById(0b55a4be-1fdc-4b91-a3a7-f964eab4522f) } result = Ok(())
5672024-05-15T03:46:24.225ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
5682024-05-15T03:46:24.225ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
5692024-05-15T03:46:24.231ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f original_time_created = 2024-05-15 03:41:43.607350 UTC target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f time_created = 2024-05-15 03:41:43.607350 UTC
5702024-05-15T03:46:24.231ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.619631ms iteration = 28
5712024-05-15T03:46:24.236ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5722024-05-15T03:46:24.236ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5732024-05-15T03:46:24.238ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5742024-05-15T03:46:24.238ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5752024-05-15T03:46:24.239ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
5762024-05-15T03:46:24.242ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 10925274-3dbb-4558-9fc7-9ec149150c20, lookup_type: ById(10925274-3dbb-4558-9fc7-9ec149150c20) } result = Ok(())
5772024-05-15T03:46:24.242ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
5782024-05-15T03:46:24.243ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
5792024-05-15T03:46:24.244ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 31.690393ms iteration = 10
5802024-05-15T03:46:24.278ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1030 zone_address = fd00:1122:3344:101::2
5812024-05-15T03:46:24.278ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
5822024-05-15T03:46:24.279ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "1cdfd95b-c096-440b-8da9-3e41fc4e9eda", "content-length": "26", "date": "Wed, 15 May 2024 03:46:24 GMT"} })
5832024-05-15T03:46:24.279ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1035 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
5842024-05-15T03:46:24.280ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1067 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
5852024-05-15T03:46:24.281ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/instance_watcher.rs:469 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0
5862024-05-15T03:46:24.281ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 68.534479ms iteration = 10
5872024-05-15T03:46:24.284ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1030 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 zone_address = fd00:1122:3344:101::2
5882024-05-15T03:46:24.284ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
5892024-05-15T03:46:24.285ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1030 zone_address = fd00:1122:3344:101::2
5902024-05-15T03:46:24.285ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
5912024-05-15T03:46:24.285ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "fc67376f-6d47-460b-b870-274e34e48613", "content-length": "26", "date": "Wed, 15 May 2024 03:46:24 GMT"} })
5922024-05-15T03:46:24.285ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1035 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
5932024-05-15T03:46:24.285ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "b8e60072-862c-4be2-99e0-5e11a2295ae0", "content-length": "26", "date": "Wed, 15 May 2024 03:46:24 GMT"} })
5942024-05-15T03:46:24.285ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1035 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
5952024-05-15T03:46:24.287ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1067 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
5962024-05-15T03:46:24.288ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1067 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4
5972024-05-15T03:46:24.367ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 156.723428ms iteration = 10
5982024-05-15T03:46:24.376ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
5992024-05-15T03:46:24.377ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "cdc76586-2274-4be1-bf41-42385fb92fe6", "content-length": "2", "date": "Wed, 15 May 2024 03:46:23 GMT"} })
6002024-05-15T03:46:24.378ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
6012024-05-15T03:46:24.379ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
6022024-05-15T03:46:24.380ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 169.205715ms iteration = 10
6032024-05-15T03:46:24.436ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
6042024-05-15T03:46:24.436ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
6052024-05-15T03:46:24.438ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:144 port = SwitchPort { id: 89eb00e0-b12a-4307-aac5-625fd364b560, rack_id: 76c05259-5403-45cc-a1c6-20f84d7e54b4, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(7f8c724a-b675-4fd5-8a78-6b7ffc7a3247) } rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0
6062024-05-15T03:46:24.438ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
6072024-05-15T03:46:24.438ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
6082024-05-15T03:46:24.447ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
6092024-05-15T03:46:24.447ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/port/qsfp0/settings", query: Some("tag=omicron"), fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "746e4054-5588-4145-b51c-72baff46a9a7", "content-length": "121", "date": "Wed, 15 May 2024 03:46:24 GMT"} })
6102024-05-15T03:46:24.447ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1643 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: None, kr: false, lane: Some(LinkId(0)), speed: Speed40G } }} } switch_port_id = "qsfp0"
6112024-05-15T03:46:24.447ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1670 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 7f8c724a-b675-4fd5-8a78-6b7ffc7a3247
6122024-05-15T03:46:24.448ZERROnexus (BackgroundTasks): unable to retrieve routes from switch background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1782 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0
6132024-05-15T03:46:24.448ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:369 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 routes = {}
6142024-05-15T03:46:24.448ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:373 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None)}}
6152024-05-15T03:46:24.448ZWARNnexus (BackgroundTasks): no discovered routes from switch. it is possible that an earlier api call failed. background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1511 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0
6162024-05-15T03:46:24.448ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:382 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 routes = {}
6172024-05-15T03:46:24.448ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:388 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 routes = {}
6182024-05-15T03:46:24.448ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:408 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4
6192024-05-15T03:46:24.448ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
6202024-05-15T03:46:24.448ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
6212024-05-15T03:46:24.450ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
6222024-05-15T03:46:24.450ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv4", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "e393f256-0605-4c91-8801-915841e46c18", "content-length": "2", "date": "Wed, 15 May 2024 03:46:24 GMT"} })
6232024-05-15T03:46:24.450ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
6242024-05-15T03:46:24.450ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv6", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "09980ea8-7bd0-438a-a905-3b3cfcfae1da", "content-length": "52", "date": "Wed, 15 May 2024 03:46:24 GMT"} })
6252024-05-15T03:46:24.450ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [V4(Ipv4Network { addr: 10.151.1.174, prefix: 24 })], port: "qsfp0" }] file = nexus/src/app/background/sync_switch_configuration.rs:457 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0
6262024-05-15T03:46:24.450ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
6272024-05-15T03:46:24.499ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/switch-ports", query: None, fragment: None }, status: 204, headers: {"x-request-id": "7b9c7727-fa59-418c-a9ec-8df7b329e40c", "date": "Wed, 15 May 2024 03:46:24 GMT"} })
6282024-05-15T03:46:24.499ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:759 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch = Switch0
6292024-05-15T03:46:24.499ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
6302024-05-15T03:46:24.500ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/network-bootstore-config", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "74a905ad-3040-4f62-9969-c1de78d29e2e", "content-length": "467", "date": "Wed, 15 May 2024 03:46:24 GMT"} })
6312024-05-15T03:46:24.500ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
6322024-05-15T03:46:24.501ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
6332024-05-15T03:46:24.504ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
6342024-05-15T03:46:24.504ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
6352024-05-15T03:46:24.507ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
6362024-05-15T03:46:24.507ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
6372024-05-15T03:46:24.508ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
6382024-05-15T03:46:24.508ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
6392024-05-15T03:46:24.515ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 302.685167ms iteration = 10
6402024-05-15T03:46:24.806ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 150 local_addr = [fd00:1122:3344:101::c]:55127 method = GET remote_addr = [fd00:1122:3344:101::d]:37120 req_id = a54aecf5-d40a-498d-a716-b6760adc2d67 response_code = 200 uri = //86c56a1d-af69-43b6-8206-e5d1baefadb2
6412024-05-15T03:46:34.211ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 29 reason = Timeout
6422024-05-15T03:46:34.211ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6432024-05-15T03:46:34.211ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6442024-05-15T03:46:34.212ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6452024-05-15T03:46:34.215ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
6462024-05-15T03:46:34.215ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6472024-05-15T03:46:34.215ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6482024-05-15T03:46:34.217ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6492024-05-15T03:46:34.217ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6502024-05-15T03:46:34.218ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6512024-05-15T03:46:34.221ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0b55a4be-1fdc-4b91-a3a7-f964eab4522f, lookup_type: ById(0b55a4be-1fdc-4b91-a3a7-f964eab4522f) } result = Ok(())
6522024-05-15T03:46:34.221ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6532024-05-15T03:46:34.221ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6542024-05-15T03:46:34.227ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f original_time_created = 2024-05-15 03:41:43.607350 UTC target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f time_created = 2024-05-15 03:41:43.607350 UTC
6552024-05-15T03:46:34.227ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 16.427598ms iteration = 29
6562024-05-15T03:46:34.806ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 147 local_addr = [fd00:1122:3344:101::c]:55127 method = GET remote_addr = [fd00:1122:3344:101::d]:37120 req_id = a15d0269-1a15-47dd-b420-65c4dde0fd3e response_code = 200 uri = //86c56a1d-af69-43b6-8206-e5d1baefadb2
6572024-05-15T03:46:44.211ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 30 reason = Timeout
6582024-05-15T03:46:44.212ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6592024-05-15T03:46:44.212ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6602024-05-15T03:46:44.214ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6612024-05-15T03:46:44.216ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
6622024-05-15T03:46:44.216ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6632024-05-15T03:46:44.216ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6642024-05-15T03:46:44.218ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6652024-05-15T03:46:44.218ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6662024-05-15T03:46:44.219ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
6672024-05-15T03:46:44.222ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0b55a4be-1fdc-4b91-a3a7-f964eab4522f, lookup_type: ById(0b55a4be-1fdc-4b91-a3a7-f964eab4522f) } result = Ok(())
6682024-05-15T03:46:44.222ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6692024-05-15T03:46:44.222ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6702024-05-15T03:46:44.228ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f original_time_created = 2024-05-15 03:41:43.607350 UTC target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f time_created = 2024-05-15 03:41:43.607350 UTC
6712024-05-15T03:46:44.228ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 16.665121ms iteration = 30
6722024-05-15T03:46:44.807ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 163 local_addr = [fd00:1122:3344:101::c]:55127 method = GET remote_addr = [fd00:1122:3344:101::d]:37120 req_id = 272ba7a5-7553-49b7-befe-66b170d64f70 response_code = 200 uri = //86c56a1d-af69-43b6-8206-e5d1baefadb2
6732024-05-15T03:46:54.211ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 31 reason = Timeout
6742024-05-15T03:46:54.211ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 6 reason = Timeout
6752024-05-15T03:46:54.211ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6762024-05-15T03:46:54.211ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 6 reason = Timeout
6772024-05-15T03:46:54.211ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
6782024-05-15T03:46:54.211ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
6792024-05-15T03:46:54.211ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 8 reason = Timeout
6802024-05-15T03:46:54.211ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 6 reason = Timeout
6812024-05-15T03:46:54.211ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
6822024-05-15T03:46:54.211ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 6 reason = Timeout
6832024-05-15T03:46:54.211ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 11 reason = Timeout
6842024-05-15T03:46:54.211ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 7 reason = Timeout
6852024-05-15T03:46:54.211ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/blueprint_execution.rs:65 reason = blueprint disabled target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f
6862024-05-15T03:46:54.211ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
6872024-05-15T03:46:54.211ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 7.14\u{b5}s iteration = 7
6882024-05-15T03:46:54.211ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 8 reason = Timeout
6892024-05-15T03:46:54.211ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 6 reason = Timeout
6902024-05-15T03:46:54.211ZWARNnexus (BackgroundTasks): Metric producer GC: statically disabled pending omicron#5284 background_task = metrics_producer_gc file = nexus/src/app/background/metrics_producer_gc.rs:43
6912024-05-15T03:46:54.211ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 3.79\u{b5}s iteration = 6
6922024-05-15T03:46:54.211ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 11 reason = Timeout
6932024-05-15T03:46:54.211ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
6942024-05-15T03:46:54.211ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 11 reason = Timeout
6952024-05-15T03:46:54.211ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:46
6962024-05-15T03:46:54.211ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 6 reason = Timeout
6972024-05-15T03:46:54.211ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
6982024-05-15T03:46:54.211ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
6992024-05-15T03:46:54.212ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7002024-05-15T03:46:54.212ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
7012024-05-15T03:46:54.212ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
7022024-05-15T03:46:54.212ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
7032024-05-15T03:46:54.212ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("3a1867b9-e5ea-4194-bfc7-0db4fb07d69e.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("9321e01e-813d-4e4f-8668-60649e9711dc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("9321e01e-813d-4e4f-8668-60649e9711dc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::8)) }], valid_until: Instant { tv_sec: 3791, tv_nsec: 638343768 } })
7042024-05-15T03:46:54.212ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
7052024-05-15T03:46:54.212ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
7062024-05-15T03:46:54.212ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("2b87d77b-ee3f-43c8-bced-dd105bbcdaa3.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("510fd65b-f60d-4faa-85be-82d26f4dd427.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("65e0e735-7f1e-479a-8127-a80c8289fcdb.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("65e0e735-7f1e-479a-8127-a80c8289fcdb.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:1::1)) }], valid_until: Instant { tv_sec: 3791, tv_nsec: 638614501 } })
7072024-05-15T03:46:54.212ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 11 reason = Timeout
7082024-05-15T03:46:54.212ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
7092024-05-15T03:46:54.212ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 11 reason = Timeout
7102024-05-15T03:46:54.212ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
7112024-05-15T03:46:54.212ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
7122024-05-15T03:46:54.212ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 11 reason = Timeout
7132024-05-15T03:46:54.212ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:44
7142024-05-15T03:46:54.212ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:48
7152024-05-15T03:46:54.212ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 5.75\u{b5}s iteration = 11
7162024-05-15T03:46:54.212ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 11 reason = Timeout
7172024-05-15T03:46:54.212ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
7182024-05-15T03:46:54.212ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
7192024-05-15T03:46:54.212ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
7202024-05-15T03:46:54.212ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
7212024-05-15T03:46:54.212ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
7222024-05-15T03:46:54.213ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 11 reason = Timeout
7232024-05-15T03:46:54.213ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
7242024-05-15T03:46:54.213ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
7252024-05-15T03:46:54.214ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
7262024-05-15T03:46:54.214ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7272024-05-15T03:46:54.215ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
7282024-05-15T03:46:54.215ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7292024-05-15T03:46:54.215ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7302024-05-15T03:46:54.215ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
7312024-05-15T03:46:54.215ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 3.753548ms iteration = 6
7322024-05-15T03:46:54.215ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7332024-05-15T03:46:54.215ZDEBGnexus (BackgroundTasks): activating background_task = service_firewall_rule_propagation iteration = 2 reason = Timeout
7342024-05-15T03:46:54.215ZDEBGnexus (service-firewall-rule-progator): starting background task for service firewall rule propagation background_task = service_firewall_rule_propagation
7352024-05-15T03:46:54.215ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7362024-05-15T03:46:54.216ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:92
7372024-05-15T03:46:54.216ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 4.424575ms iteration = 11
7382024-05-15T03:46:54.216ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7392024-05-15T03:46:54.216ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7402024-05-15T03:46:54.216ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7412024-05-15T03:46:54.216ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7422024-05-15T03:46:54.217ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7432024-05-15T03:46:54.218ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
7442024-05-15T03:46:54.218ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 6.487116ms iteration = 6
7452024-05-15T03:46:54.218ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3791, tv_nsec: 644397850 } })
7462024-05-15T03:46:54.218ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7472024-05-15T03:46:54.219ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7482024-05-15T03:46:54.220ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
7492024-05-15T03:46:54.220ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
7502024-05-15T03:46:54.221ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
7512024-05-15T03:46:54.221ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3791, tv_nsec: 647535132 } })
7522024-05-15T03:46:54.221ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
7532024-05-15T03:46:54.221ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
7542024-05-15T03:46:54.221ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
7552024-05-15T03:46:54.221ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
7562024-05-15T03:46:54.221ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
7572024-05-15T03:46:54.222ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
7582024-05-15T03:46:54.222ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
7592024-05-15T03:46:54.222ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7602024-05-15T03:46:54.223ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
7612024-05-15T03:46:54.223ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
7622024-05-15T03:46:54.223ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7632024-05-15T03:46:54.223ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
7642024-05-15T03:46:54.223ZWARNnexus (BackgroundTasks): physical disk adoption task started background_task = physical_disk_adoption file = nexus/src/app/background/physical_disk_adoption.rs:100
7652024-05-15T03:46:54.223ZWARNnexus (BackgroundTasks): Physical Disk Adoption: skipped background_task = physical_disk_adoption file = nexus/src/app/background/physical_disk_adoption.rs:104 reason = no inventory
7662024-05-15T03:46:54.223ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 11.264714ms iteration = 11
7672024-05-15T03:46:54.223ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
7682024-05-15T03:46:54.223ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7692024-05-15T03:46:54.224ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7702024-05-15T03:46:54.224ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
7712024-05-15T03:46:54.224ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
7722024-05-15T03:46:54.224ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
7732024-05-15T03:46:54.224ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
7742024-05-15T03:46:54.224ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
7752024-05-15T03:46:54.224ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
7762024-05-15T03:46:54.224ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
7772024-05-15T03:46:54.224ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
7782024-05-15T03:46:54.225ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7792024-05-15T03:46:54.225ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7802024-05-15T03:46:54.225ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
7812024-05-15T03:46:54.226ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
7822024-05-15T03:46:54.226ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
7832024-05-15T03:46:54.226ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
7842024-05-15T03:46:54.226ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7852024-05-15T03:46:54.226ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
7862024-05-15T03:46:54.226ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
7872024-05-15T03:46:54.226ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7882024-05-15T03:46:54.226ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
7892024-05-15T03:46:54.227ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7902024-05-15T03:46:54.227ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
7912024-05-15T03:46:54.227ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
7922024-05-15T03:46:54.227ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
7932024-05-15T03:46:54.227ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
7942024-05-15T03:46:54.227ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
7952024-05-15T03:46:54.227ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7962024-05-15T03:46:54.227ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1022
7972024-05-15T03:46:54.228ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
7982024-05-15T03:46:54.228ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7992024-05-15T03:46:54.228ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8002024-05-15T03:46:54.228ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8012024-05-15T03:46:54.230ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8022024-05-15T03:46:54.231ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1022
8032024-05-15T03:46:54.231ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
8042024-05-15T03:46:54.231ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3791, tv_nsec: 657936468 } })
8052024-05-15T03:46:54.231ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
8062024-05-15T03:46:54.232ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
8072024-05-15T03:46:54.234ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
8082024-05-15T03:46:54.234ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 52cc15d1-4ce1-4e38-8c6c-f9f099a8a5be dns_zone_name = oxide.test found_so_far = 0 version = 2
8092024-05-15T03:46:54.234ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
8102024-05-15T03:46:54.234ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
8112024-05-15T03:46:54.234ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
8122024-05-15T03:46:54.234ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
8132024-05-15T03:46:54.234ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
8142024-05-15T03:46:54.235ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0b55a4be-1fdc-4b91-a3a7-f964eab4522f, lookup_type: ById(0b55a4be-1fdc-4b91-a3a7-f964eab4522f) } result = Ok(())
8152024-05-15T03:46:54.235ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8162024-05-15T03:46:54.235ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
8172024-05-15T03:46:54.235ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
8182024-05-15T03:46:54.235ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8192024-05-15T03:46:54.236ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8202024-05-15T03:46:54.236ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
8212024-05-15T03:46:54.236ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
8222024-05-15T03:46:54.236ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
8232024-05-15T03:46:54.236ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8242024-05-15T03:46:54.236ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1022 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4
8252024-05-15T03:46:54.236ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
8262024-05-15T03:46:54.237ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8272024-05-15T03:46:54.239ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8282024-05-15T03:46:54.239ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8292024-05-15T03:46:54.239ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8302024-05-15T03:46:54.240ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
8312024-05-15T03:46:54.240ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
8322024-05-15T03:46:54.241ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
8332024-05-15T03:46:54.241ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8342024-05-15T03:46:54.241ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8352024-05-15T03:46:54.241ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
8362024-05-15T03:46:54.241ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 3698e770-e01a-43e5-856e-b62ced4fb671 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
8372024-05-15T03:46:54.241ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
8382024-05-15T03:46:54.241ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
8392024-05-15T03:46:54.242ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
8402024-05-15T03:46:54.242ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
8412024-05-15T03:46:54.242ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
8422024-05-15T03:46:54.242ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8432024-05-15T03:46:54.242ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 52cc15d1-4ce1-4e38-8c6c-f9f099a8a5be dns_zone_name = oxide.test found_so_far = 1 version = 2
8442024-05-15T03:46:54.242ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
8452024-05-15T03:46:54.243ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2024-05-15 03:41:57.099854 UTC dns_group = external generation = 2
8462024-05-15T03:46:54.243ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 31.395501ms iteration = 6
8472024-05-15T03:46:54.243ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8482024-05-15T03:46:54.244ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f original_time_created = 2024-05-15 03:41:43.607350 UTC target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f time_created = 2024-05-15 03:41:43.607350 UTC
8492024-05-15T03:46:54.244ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 32.971306ms iteration = 31
8502024-05-15T03:46:54.244ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
8512024-05-15T03:46:54.244ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
8522024-05-15T03:46:54.246ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
8532024-05-15T03:46:54.247ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
8542024-05-15T03:46:54.247ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
8552024-05-15T03:46:54.247ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
8562024-05-15T03:46:54.248ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
8572024-05-15T03:46:54.248ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8582024-05-15T03:46:54.248ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
8592024-05-15T03:46:54.248ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8602024-05-15T03:46:54.249ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8612024-05-15T03:46:54.250ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 3698e770-e01a-43e5-856e-b62ced4fb671 dns_zone_name = control-plane.oxide.internal found_so_far = 49 version = 1
8622024-05-15T03:46:54.250ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
8632024-05-15T03:46:54.250ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2024-05-15 03:41:56.187733 UTC dns_group = internal generation = 1
8642024-05-15T03:46:54.250ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 38.381851ms iteration = 6
8652024-05-15T03:46:54.251ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 10925274-3dbb-4558-9fc7-9ec149150c20, lookup_type: ById(10925274-3dbb-4558-9fc7-9ec149150c20) } result = Ok(())
8662024-05-15T03:46:54.251ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
8672024-05-15T03:46:54.251ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
8682024-05-15T03:46:54.252ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 40.21035ms iteration = 11
8692024-05-15T03:46:54.253ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8702024-05-15T03:46:54.253ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8712024-05-15T03:46:54.263ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8722024-05-15T03:46:54.263ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8732024-05-15T03:46:54.269ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8742024-05-15T03:46:54.269ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8752024-05-15T03:46:54.271ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8762024-05-15T03:46:54.271ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8772024-05-15T03:46:54.272ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8782024-05-15T03:46:54.273ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8792024-05-15T03:46:54.274ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8802024-05-15T03:46:54.274ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8812024-05-15T03:46:54.275ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8822024-05-15T03:46:54.279ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
8832024-05-15T03:46:54.279ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8842024-05-15T03:46:54.279ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8852024-05-15T03:46:54.280ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8862024-05-15T03:46:54.280ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8872024-05-15T03:46:54.282ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8882024-05-15T03:46:54.282ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8892024-05-15T03:46:54.284ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8902024-05-15T03:46:54.284ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8912024-05-15T03:46:54.288ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8922024-05-15T03:46:54.288ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8932024-05-15T03:46:54.289ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8942024-05-15T03:46:54.290ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8952024-05-15T03:46:54.291ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
8962024-05-15T03:46:54.291ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
8972024-05-15T03:46:54.291ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2024-05-15T03:41:54.396604Z, time_modified: 2024-05-15T03:41:54.396604Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 39f7e4ba-533e-4258-9f79-1873f513f101, db_silo: Silo { identity: SiloIdentity { id: 39f7e4ba-533e-4258-9f79-1873f513f101, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-05-15T03:41:57.052187Z, time_modified: 2024-05-15T03:41:57.052187Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "370e7f12febb2dc0abbebc4192727b23c734c770fdab3368ed3fd759491f6fed" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 39f7e4ba-533e-4258-9f79-1873f513f101, db_silo: Silo { identity: SiloIdentity { id: 39f7e4ba-533e-4258-9f79-1873f513f101, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-05-15T03:41:57.052187Z, time_modified: 2024-05-15T03:41:57.052187Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "370e7f12febb2dc0abbebc4192727b23c734c770fdab3368ed3fd759491f6fed" }] }) }
8982024-05-15T03:46:54.291ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 80.290619ms iteration = 6
8992024-05-15T03:46:54.292ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9002024-05-15T03:46:54.296ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
9012024-05-15T03:46:54.296ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9022024-05-15T03:46:54.296ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9032024-05-15T03:46:54.297ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9042024-05-15T03:46:54.297ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9052024-05-15T03:46:54.298ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9062024-05-15T03:46:54.299ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9072024-05-15T03:46:54.300ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9082024-05-15T03:46:54.303ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
9092024-05-15T03:46:54.303ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9102024-05-15T03:46:54.303ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9112024-05-15T03:46:54.306ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/instance_watcher.rs:469 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0
9122024-05-15T03:46:54.306ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 92.637795ms iteration = 11
9132024-05-15T03:46:54.364ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1030 zone_address = fd00:1122:3344:101::2
9142024-05-15T03:46:54.364ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
9152024-05-15T03:46:54.364ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "1a1ac029-4260-4cb6-91fd-1bf585a18120", "content-length": "26", "date": "Wed, 15 May 2024 03:46:54 GMT"} })
9162024-05-15T03:46:54.364ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1035 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
9172024-05-15T03:46:54.366ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1067 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
9182024-05-15T03:46:54.397ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1030 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 zone_address = fd00:1122:3344:101::2
9192024-05-15T03:46:54.397ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
9202024-05-15T03:46:54.398ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "2b5714eb-be68-464f-bced-b5959c53c0cb", "content-length": "26", "date": "Wed, 15 May 2024 03:46:54 GMT"} })
9212024-05-15T03:46:54.398ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1035 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
9222024-05-15T03:46:54.399ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9232024-05-15T03:46:54.400ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1067 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4
9242024-05-15T03:46:54.400ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9252024-05-15T03:46:54.401ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9262024-05-15T03:46:54.402ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9272024-05-15T03:46:54.403ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9282024-05-15T03:46:54.403ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9292024-05-15T03:46:54.405ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9302024-05-15T03:46:54.405ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9312024-05-15T03:46:54.407ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9322024-05-15T03:46:54.407ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9332024-05-15T03:46:54.408ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9342024-05-15T03:46:54.409ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9352024-05-15T03:46:54.410ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9362024-05-15T03:46:54.410ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9372024-05-15T03:46:54.411ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9382024-05-15T03:46:54.416ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
9392024-05-15T03:46:54.416ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9402024-05-15T03:46:54.416ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9412024-05-15T03:46:54.417ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9422024-05-15T03:46:54.417ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9432024-05-15T03:46:54.418ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9442024-05-15T03:46:54.418ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9452024-05-15T03:46:54.419ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9462024-05-15T03:46:54.422ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1030 zone_address = fd00:1122:3344:101::2
9472024-05-15T03:46:54.422ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
9482024-05-15T03:46:54.422ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "a9a1a642-62f6-426c-a294-7cd332c26ce2", "content-length": "26", "date": "Wed, 15 May 2024 03:46:54 GMT"} })
9492024-05-15T03:46:54.422ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1035 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
9502024-05-15T03:46:54.423ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
9512024-05-15T03:46:54.423ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9522024-05-15T03:46:54.424ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9532024-05-15T03:46:54.424ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1067 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
9542024-05-15T03:46:54.437ZDEBGnexus (service-firewall-rule-progator): resolved names for firewall rules background_task = service_firewall_rule_propagation instance_interfaces = {} subnet_interfaces = {Name("external-dns"): [NetworkInterface { id: 4de9d359-2f72-48ba-aa12-b0ed7f080dfb, kind: Service { id: 3a1867b9-e5ea-4194-bfc7-0db4fb07d69e }, name: Name("external-dns-3a1867b9-e5ea-4194-bfc7-0db4fb07d69e"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 232, 42])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: 7b6a5ce1-fd04-40ca-ac0a-846332fe2039, kind: Service { id: 9321e01e-813d-4e4f-8668-60649e9711dc }, name: Name("external-dns-9321e01e-813d-4e4f-8668-60649e9711dc"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 177, 26])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }], Name("nexus"): [NetworkInterface { id: 69799215-3743-44c5-8f49-5eabb2ad0fb7, kind: Service { id: 86c56a1d-af69-43b6-8206-e5d1baefadb2 }, name: Name("nexus-86c56a1d-af69-43b6-8206-e5d1baefadb2"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 180, 110])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: c64026ce-be7e-46ce-b478-436df03885ee, kind: Service { id: 322e1a51-ecb0-41b8-a4d2-d7336940b999 }, name: Name("nexus-322e1a51-ecb0-41b8-a4d2-d7336940b999"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 166, 163])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: fc45936f-53c7-4d73-9f08-a9e09a4d464f, kind: Service { id: 28f40080-7678-4a97-8f4a-5026eb3b1dc3 }, name: Name("nexus-28f40080-7678-4a97-8f4a-5026eb3b1dc3"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 251, 155])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }]} subnet_networks = {Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })], Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })]} vpc_interfaces = {}
9552024-05-15T03:46:54.437ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9562024-05-15T03:46:54.438ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9572024-05-15T03:46:54.440ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9582024-05-15T03:46:54.442ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Fleet result = Ok(())
9592024-05-15T03:46:54.442ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9602024-05-15T03:46:54.442ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9612024-05-15T03:46:54.447ZDEBGnexus (service-firewall-rule-progator): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified. background_task = service_firewall_rule_propagation
9622024-05-15T03:46:54.447ZDEBGnexus (service-firewall-rule-progator): resolved firewall rules for sled agents background_task = service_firewall_rule_propagation sled_agent_rules = [VpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("53")]), filter_protocols: Some([Udp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 4de9d359-2f72-48ba-aa12-b0ed7f080dfb, kind: Service { id: 3a1867b9-e5ea-4194-bfc7-0db4fb07d69e }, name: Name("external-dns-3a1867b9-e5ea-4194-bfc7-0db4fb07d69e"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 232, 42])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: 7b6a5ce1-fd04-40ca-ac0a-846332fe2039, kind: Service { id: 9321e01e-813d-4e4f-8668-60649e9711dc }, name: Name("external-dns-9321e01e-813d-4e4f-8668-60649e9711dc"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 177, 26])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }] }, VpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("80"), L4PortRange("443")]), filter_protocols: Some([Tcp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 69799215-3743-44c5-8f49-5eabb2ad0fb7, kind: Service { id: 86c56a1d-af69-43b6-8206-e5d1baefadb2 }, name: Name("nexus-86c56a1d-af69-43b6-8206-e5d1baefadb2"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 180, 110])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: c64026ce-be7e-46ce-b478-436df03885ee, kind: Service { id: 322e1a51-ecb0-41b8-a4d2-d7336940b999 }, name: Name("nexus-322e1a51-ecb0-41b8-a4d2-d7336940b999"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 166, 163])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: fc45936f-53c7-4d73-9f08-a9e09a4d464f, kind: Service { id: 28f40080-7678-4a97-8f4a-5026eb3b1dc3 }, name: Name("nexus-28f40080-7678-4a97-8f4a-5026eb3b1dc3"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 251, 155])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }] }]
9632024-05-15T03:46:54.447ZDEBGnexus (service-firewall-rule-progator): resolved 2 rules for sleds background_task = service_firewall_rule_propagation
9642024-05-15T03:46:54.484ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
9652024-05-15T03:46:54.494ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
9662024-05-15T03:46:54.539ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
9672024-05-15T03:46:54.540ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "a419b487-34d0-4ae8-98c4-2d7f148d6fe9", "content-length": "2", "date": "Wed, 15 May 2024 03:46:54 GMT"} })
9682024-05-15T03:46:54.541ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
9692024-05-15T03:46:54.541ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
9702024-05-15T03:46:54.542ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 330.978296ms iteration = 11
9712024-05-15T03:46:54.600ZDEBGnexus (service-firewall-rule-progator): resolved sleds for vpc oxide-services background_task = service_firewall_rule_propagation vpc_to_sled = [Sled { identity: SledIdentity { id: 10925274-3dbb-4558-9fc7-9ec149150c20, time_created: 2024-05-15T03:41:55.484234Z, time_modified: 2024-05-15T03:41:55.484234Z }, time_deleted: None, rcgen: Generation(Generation(19)), rack_id: 76c05259-5403-45cc-a1c6-20f84d7e54b4, is_scrimlet: true, serial_number: "buskin", part_number: "i86pc", revision: 0, usable_hardware_threads: SqlU32(48), usable_physical_ram: ByteCount(ByteCount(102952026112)), reservoir_size: ByteCount(ByteCount(51474595840)), ip: fd00:1122:3344:101::1, port: SqlU16(12345), last_used_address: fd00:1122:3344:101::ffff, policy: InService, state: Active, sled_agent_gen: Generation(Generation(1)) }]
9722024-05-15T03:46:54.600ZDEBGnexus (service-firewall-rule-progator): sending firewall rules to sled agents background_task = service_firewall_rule_propagation
9732024-05-15T03:46:54.600ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9742024-05-15T03:46:54.601ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9752024-05-15T03:46:54.602ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9762024-05-15T03:46:54.602ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9772024-05-15T03:46:54.604ZDEBGnexus (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9782024-05-15T03:46:54.606ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Sled { parent: Fleet, key: 10925274-3dbb-4558-9fc7-9ec149150c20, lookup_type: ById(10925274-3dbb-4558-9fc7-9ec149150c20) } result = Ok(())
9792024-05-15T03:46:54.687ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 475.42491ms iteration = 11
9802024-05-15T03:46:54.736ZDEBGnexus (service-firewall-rule-progator): client request SledAgent = 10925274-3dbb-4558-9fc7-9ec149150c20 background_task = service_firewall_rule_propagation body = Some(Body) method = PUT uri = http://[fd00:1122:3344:101::1]:12345/vpc/001de000-074c-4000-8000-000000000000/firewall/rules
9812024-05-15T03:46:54.737ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
9822024-05-15T03:46:54.760ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
9832024-05-15T03:46:54.760ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::8)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "0b73bbca-7d94-4b29-97e7-69025eb30d98", "date": "Wed, 15 May 2024 03:46:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
9842024-05-15T03:46:54.760ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
9852024-05-15T03:46:54.761ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::9)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "22ee03eb-4028-4c16-91f9-fe8516a213d0", "date": "Wed, 15 May 2024 03:46:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
9862024-05-15T03:46:54.761ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
9872024-05-15T03:46:54.762ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
9882024-05-15T03:46:54.762ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 551.273974ms iteration = 8
9892024-05-15T03:46:54.768ZDEBGnexus (service-firewall-rule-progator): client response SledAgent = 10925274-3dbb-4558-9fc7-9ec149150c20 background_task = service_firewall_rule_propagation result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/vpc/001de000-074c-4000-8000-000000000000/firewall/rules", query: None, fragment: None }, status: 204, headers: {"x-request-id": "7a138a16-eecd-448f-9dad-adc7290f285e", "date": "Wed, 15 May 2024 03:46:54 GMT"} })
9902024-05-15T03:46:54.770ZINFOnexus (service-firewall-rule-progator): updated firewall rules on 1 sleds background_task = service_firewall_rule_propagation file = nexus/networking/src/firewall_rules.rs:486
9912024-05-15T03:46:54.770ZDEBGnexus (service-firewall-rule-progator): successfully propagated service firewall rules background_task = service_firewall_rule_propagation elapsed = 554.779649ms
9922024-05-15T03:46:54.770ZDEBGnexus (BackgroundTasks): activation complete background_task = service_firewall_rule_propagation elapsed = 554.79082ms iteration = 2
9932024-05-15T03:46:54.807ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 147 local_addr = [fd00:1122:3344:101::c]:55127 method = GET remote_addr = [fd00:1122:3344:101::d]:37120 req_id = 38007c53-c43f-4989-b4b9-6aef308fd8cd response_code = 200 uri = //86c56a1d-af69-43b6-8206-e5d1baefadb2
9942024-05-15T03:46:54.815ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
9952024-05-15T03:46:54.815ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:2::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "7b21c600-f84d-4a15-be9c-b824d5eeae8a", "date": "Wed, 15 May 2024 03:46:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
9962024-05-15T03:46:54.815ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
9972024-05-15T03:46:54.816ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:3::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "915c5a34-42d5-443d-8b1f-6a405d19ceb8", "date": "Wed, 15 May 2024 03:46:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
9982024-05-15T03:46:54.816ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
9992024-05-15T03:46:54.818ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:1::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "163494f9-01e5-41e9-9ed6-a2f5c288fbf5", "date": "Wed, 15 May 2024 03:46:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
10002024-05-15T03:46:54.818ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
10012024-05-15T03:46:54.820ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
10022024-05-15T03:46:54.820ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 608.869601ms iteration = 8
10032024-05-15T03:46:54.825ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
10042024-05-15T03:46:54.825ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
10052024-05-15T03:46:54.827ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:144 port = SwitchPort { id: 89eb00e0-b12a-4307-aac5-625fd364b560, rack_id: 76c05259-5403-45cc-a1c6-20f84d7e54b4, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(7f8c724a-b675-4fd5-8a78-6b7ffc7a3247) } rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0
10062024-05-15T03:46:54.827ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
10072024-05-15T03:46:54.827ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
10082024-05-15T03:46:54.835ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
10092024-05-15T03:46:54.836ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/port/qsfp0/settings", query: Some("tag=omicron"), fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "9a5fc2ed-8e4a-43af-ba44-7ffce6f37dcf", "content-length": "121", "date": "Wed, 15 May 2024 03:46:54 GMT"} })
10102024-05-15T03:46:54.836ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1643 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: None, kr: false, lane: Some(LinkId(0)), speed: Speed40G } }} } switch_port_id = "qsfp0"
10112024-05-15T03:46:54.836ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1670 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 7f8c724a-b675-4fd5-8a78-6b7ffc7a3247
10122024-05-15T03:46:54.836ZERROnexus (BackgroundTasks): unable to retrieve routes from switch background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1782 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0
10132024-05-15T03:46:54.836ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:369 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 routes = {}
10142024-05-15T03:46:54.836ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:373 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None)}}
10152024-05-15T03:46:54.836ZWARNnexus (BackgroundTasks): no discovered routes from switch. it is possible that an earlier api call failed. background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1511 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0
10162024-05-15T03:46:54.836ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:382 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 routes = {}
10172024-05-15T03:46:54.836ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:388 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 routes = {}
10182024-05-15T03:46:54.836ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:408 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4
10192024-05-15T03:46:54.836ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
10202024-05-15T03:46:54.836ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
10212024-05-15T03:46:54.837ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
10222024-05-15T03:46:54.838ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv4", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "561d891c-6e91-4977-971c-17e0999e65d5", "content-length": "2", "date": "Wed, 15 May 2024 03:46:54 GMT"} })
10232024-05-15T03:46:54.838ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
10242024-05-15T03:46:54.838ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv6", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "250434dc-586d-4477-98f1-4fc8ebc75651", "content-length": "52", "date": "Wed, 15 May 2024 03:46:54 GMT"} })
10252024-05-15T03:46:54.838ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [V4(Ipv4Network { addr: 10.151.1.174, prefix: 24 })], port: "qsfp0" }] file = nexus/src/app/background/sync_switch_configuration.rs:457 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0
10262024-05-15T03:46:54.838ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
10272024-05-15T03:46:54.883ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/switch-ports", query: None, fragment: None }, status: 204, headers: {"x-request-id": "89d666b4-bc6a-435c-bd2f-54edf9ba5103", "date": "Wed, 15 May 2024 03:46:54 GMT"} })
10282024-05-15T03:46:54.883ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:759 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch = Switch0
10292024-05-15T03:46:54.883ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
10302024-05-15T03:46:54.884ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/network-bootstore-config", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "46237d55-bec5-42a0-997e-349baa928e60", "content-length": "467", "date": "Wed, 15 May 2024 03:46:54 GMT"} })
10312024-05-15T03:46:54.884ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
10322024-05-15T03:46:54.884ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
10332024-05-15T03:46:54.886ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
10342024-05-15T03:46:54.887ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
10352024-05-15T03:46:54.889ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
10362024-05-15T03:46:54.889ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
10372024-05-15T03:46:54.890ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
10382024-05-15T03:46:54.890ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
10392024-05-15T03:46:54.897ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 684.777626ms iteration = 11
10402024-05-15T03:47:04.212ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 32 reason = Timeout
10412024-05-15T03:47:04.212ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10422024-05-15T03:47:04.212ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10432024-05-15T03:47:04.214ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10442024-05-15T03:47:04.216ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
10452024-05-15T03:47:04.216ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10462024-05-15T03:47:04.217ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10472024-05-15T03:47:04.218ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10482024-05-15T03:47:04.218ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10492024-05-15T03:47:04.219ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10502024-05-15T03:47:04.222ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0b55a4be-1fdc-4b91-a3a7-f964eab4522f, lookup_type: ById(0b55a4be-1fdc-4b91-a3a7-f964eab4522f) } result = Ok(())
10512024-05-15T03:47:04.222ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10522024-05-15T03:47:04.222ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10532024-05-15T03:47:04.228ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f original_time_created = 2024-05-15 03:41:43.607350 UTC target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f time_created = 2024-05-15 03:41:43.607350 UTC
10542024-05-15T03:47:04.228ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 16.203255ms iteration = 32
10552024-05-15T03:47:04.807ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 161 local_addr = [fd00:1122:3344:101::c]:55127 method = GET remote_addr = [fd00:1122:3344:101::d]:37120 req_id = cc3274cc-2a2d-43d5-a9d5-4bbba661d2f7 response_code = 200 uri = //86c56a1d-af69-43b6-8206-e5d1baefadb2
10562024-05-15T03:47:14.212ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 33 reason = Timeout
10572024-05-15T03:47:14.212ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10582024-05-15T03:47:14.212ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10592024-05-15T03:47:14.214ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10602024-05-15T03:47:14.216ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
10612024-05-15T03:47:14.216ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10622024-05-15T03:47:14.216ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10632024-05-15T03:47:14.218ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10642024-05-15T03:47:14.218ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10652024-05-15T03:47:14.219ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10662024-05-15T03:47:14.221ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0b55a4be-1fdc-4b91-a3a7-f964eab4522f, lookup_type: ById(0b55a4be-1fdc-4b91-a3a7-f964eab4522f) } result = Ok(())
10672024-05-15T03:47:14.221ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10682024-05-15T03:47:14.222ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10692024-05-15T03:47:14.232ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f original_time_created = 2024-05-15 03:41:43.607350 UTC target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f time_created = 2024-05-15 03:41:43.607350 UTC
10702024-05-15T03:47:14.232ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 19.703441ms iteration = 33
10712024-05-15T03:47:14.807ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 143 local_addr = [fd00:1122:3344:101::c]:55127 method = GET remote_addr = [fd00:1122:3344:101::d]:37120 req_id = c2f7817c-fab0-4a64-a86a-8bd7145b6b94 response_code = 200 uri = //86c56a1d-af69-43b6-8206-e5d1baefadb2
10722024-05-15T03:47:24.212ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 34 reason = Timeout
10732024-05-15T03:47:24.212ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
10742024-05-15T03:47:24.212ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 12 reason = Timeout
10752024-05-15T03:47:24.212ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 12 reason = Timeout
10762024-05-15T03:47:24.212ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
10772024-05-15T03:47:24.212ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
10782024-05-15T03:47:24.212ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 12 reason = Timeout
10792024-05-15T03:47:24.212ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:46
10802024-05-15T03:47:24.212ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
10812024-05-15T03:47:24.212ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
10822024-05-15T03:47:24.212ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
10832024-05-15T03:47:24.213ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 12 reason = Timeout
10842024-05-15T03:47:24.213ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 12 reason = Timeout
10852024-05-15T03:47:24.213ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
10862024-05-15T03:47:24.213ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
10872024-05-15T03:47:24.213ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 12 reason = Timeout
10882024-05-15T03:47:24.213ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 12 reason = Timeout
10892024-05-15T03:47:24.213ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:44
10902024-05-15T03:47:24.213ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:48
10912024-05-15T03:47:24.213ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
10922024-05-15T03:47:24.213ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 5.03\u{b5}s iteration = 12
10932024-05-15T03:47:24.213ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
10942024-05-15T03:47:24.213ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
10952024-05-15T03:47:24.213ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
10962024-05-15T03:47:24.214ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 12 reason = Timeout
10972024-05-15T03:47:24.214ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
10982024-05-15T03:47:24.214ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
10992024-05-15T03:47:24.214ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3821, tv_nsec: 640463360 } })
11002024-05-15T03:47:24.215ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11012024-05-15T03:47:24.215ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11022024-05-15T03:47:24.215ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11032024-05-15T03:47:24.215ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:92
11042024-05-15T03:47:24.215ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.385164ms iteration = 12
11052024-05-15T03:47:24.215ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11062024-05-15T03:47:24.216ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3821, tv_nsec: 641657672 } })
11072024-05-15T03:47:24.216ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11082024-05-15T03:47:24.216ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1022
11092024-05-15T03:47:24.219ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
11102024-05-15T03:47:24.219ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
11112024-05-15T03:47:24.220ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
11122024-05-15T03:47:24.220ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1022
11132024-05-15T03:47:24.220ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
11142024-05-15T03:47:24.220ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
11152024-05-15T03:47:24.220ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
11162024-05-15T03:47:24.220ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
11172024-05-15T03:47:24.220ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
11182024-05-15T03:47:24.220ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
11192024-05-15T03:47:24.220ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
11202024-05-15T03:47:24.220ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
11212024-05-15T03:47:24.220ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
11222024-05-15T03:47:24.221ZWARNnexus (BackgroundTasks): physical disk adoption task started background_task = physical_disk_adoption file = nexus/src/app/background/physical_disk_adoption.rs:100
11232024-05-15T03:47:24.221ZWARNnexus (BackgroundTasks): Physical Disk Adoption: skipped background_task = physical_disk_adoption file = nexus/src/app/background/physical_disk_adoption.rs:104 reason = no inventory
11242024-05-15T03:47:24.221ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 8.405196ms iteration = 12
11252024-05-15T03:47:24.222ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
11262024-05-15T03:47:24.222ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
11272024-05-15T03:47:24.222ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
11282024-05-15T03:47:24.222ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
11292024-05-15T03:47:24.222ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
11302024-05-15T03:47:24.224ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11312024-05-15T03:47:24.224ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
11322024-05-15T03:47:24.224ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
11332024-05-15T03:47:24.224ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3821, tv_nsec: 650292210 } })
11342024-05-15T03:47:24.224ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/instance_watcher.rs:469 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0
11352024-05-15T03:47:24.224ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 10.592337ms iteration = 12
11362024-05-15T03:47:24.226ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1022 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4
11372024-05-15T03:47:24.228ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0b55a4be-1fdc-4b91-a3a7-f964eab4522f, lookup_type: ById(0b55a4be-1fdc-4b91-a3a7-f964eab4522f) } result = Ok(())
11382024-05-15T03:47:24.228ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
11392024-05-15T03:47:24.229ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
11402024-05-15T03:47:24.236ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f original_time_created = 2024-05-15 03:41:43.607350 UTC target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f time_created = 2024-05-15 03:41:43.607350 UTC
11412024-05-15T03:47:24.236ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 24.202577ms iteration = 34
11422024-05-15T03:47:24.255ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
11432024-05-15T03:47:24.255ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
11442024-05-15T03:47:24.256ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
11452024-05-15T03:47:24.256ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
11462024-05-15T03:47:24.257ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11472024-05-15T03:47:24.260ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 10925274-3dbb-4558-9fc7-9ec149150c20, lookup_type: ById(10925274-3dbb-4558-9fc7-9ec149150c20) } result = Ok(())
11482024-05-15T03:47:24.260ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
11492024-05-15T03:47:24.260ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
11502024-05-15T03:47:24.261ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 48.371143ms iteration = 12
11512024-05-15T03:47:24.313ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1030 zone_address = fd00:1122:3344:101::2
11522024-05-15T03:47:24.313ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
11532024-05-15T03:47:24.313ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "2ee7827d-b2a7-405d-ae1a-2ff09ffd8598", "content-length": "26", "date": "Wed, 15 May 2024 03:47:24 GMT"} })
11542024-05-15T03:47:24.313ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1035 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
11552024-05-15T03:47:24.315ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1067 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
11562024-05-15T03:47:24.315ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1030 zone_address = fd00:1122:3344:101::2
11572024-05-15T03:47:24.315ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
11582024-05-15T03:47:24.316ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "fe009763-4de5-41e2-a781-803304a1c9d8", "content-length": "26", "date": "Wed, 15 May 2024 03:47:24 GMT"} })
11592024-05-15T03:47:24.316ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1035 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
11602024-05-15T03:47:24.316ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1030 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 zone_address = fd00:1122:3344:101::2
11612024-05-15T03:47:24.316ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
11622024-05-15T03:47:24.317ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "e69ce863-9ee2-498a-90d4-d4735dd3972f", "content-length": "26", "date": "Wed, 15 May 2024 03:47:24 GMT"} })
11632024-05-15T03:47:24.317ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1035 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
11642024-05-15T03:47:24.317ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1067 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
11652024-05-15T03:47:24.318ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1067 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4
11662024-05-15T03:47:24.388ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 176.705522ms iteration = 12
11672024-05-15T03:47:24.394ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
11682024-05-15T03:47:24.395ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "d5efc727-63aa-4d0b-947e-d802bb5247aa", "content-length": "2", "date": "Wed, 15 May 2024 03:47:23 GMT"} })
11692024-05-15T03:47:24.396ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
11702024-05-15T03:47:24.396ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
11712024-05-15T03:47:24.398ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 186.164719ms iteration = 12
11722024-05-15T03:47:24.451ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
11732024-05-15T03:47:24.451ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
11742024-05-15T03:47:24.453ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:144 port = SwitchPort { id: 89eb00e0-b12a-4307-aac5-625fd364b560, rack_id: 76c05259-5403-45cc-a1c6-20f84d7e54b4, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(7f8c724a-b675-4fd5-8a78-6b7ffc7a3247) } rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0
11752024-05-15T03:47:24.453ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
11762024-05-15T03:47:24.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
11772024-05-15T03:47:24.459ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
11782024-05-15T03:47:24.460ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/port/qsfp0/settings", query: Some("tag=omicron"), fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "3acbc772-0db0-4ef1-80a0-a797ccc57f6b", "content-length": "121", "date": "Wed, 15 May 2024 03:47:23 GMT"} })
11792024-05-15T03:47:24.460ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1643 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: None, kr: false, lane: Some(LinkId(0)), speed: Speed40G } }} } switch_port_id = "qsfp0"
11802024-05-15T03:47:24.460ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1670 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 7f8c724a-b675-4fd5-8a78-6b7ffc7a3247
11812024-05-15T03:47:24.461ZERROnexus (BackgroundTasks): unable to retrieve routes from switch background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1782 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0
11822024-05-15T03:47:24.461ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:369 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 routes = {}
11832024-05-15T03:47:24.461ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:373 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None)}}
11842024-05-15T03:47:24.461ZWARNnexus (BackgroundTasks): no discovered routes from switch. it is possible that an earlier api call failed. background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1511 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0
11852024-05-15T03:47:24.461ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:382 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 routes = {}
11862024-05-15T03:47:24.461ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:388 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 routes = {}
11872024-05-15T03:47:24.461ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:408 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4
11882024-05-15T03:47:24.461ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
11892024-05-15T03:47:24.461ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
11902024-05-15T03:47:24.463ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
11912024-05-15T03:47:24.463ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv4", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "0ed57f16-f19a-4a7d-95e9-483fa29181e7", "content-length": "2", "date": "Wed, 15 May 2024 03:47:23 GMT"} })
11922024-05-15T03:47:24.463ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
11932024-05-15T03:47:24.463ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv6", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "4c0f7ead-6496-4967-a0d4-43aeecde53a0", "content-length": "52", "date": "Wed, 15 May 2024 03:47:23 GMT"} })
11942024-05-15T03:47:24.463ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [V4(Ipv4Network { addr: 10.151.1.174, prefix: 24 })], port: "qsfp0" }] file = nexus/src/app/background/sync_switch_configuration.rs:457 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0
11952024-05-15T03:47:24.463ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
11962024-05-15T03:47:24.514ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/switch-ports", query: None, fragment: None }, status: 204, headers: {"x-request-id": "3689ff6b-746a-492d-a745-71a3a6b73e5b", "date": "Wed, 15 May 2024 03:47:24 GMT"} })
11972024-05-15T03:47:24.514ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:759 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch = Switch0
11982024-05-15T03:47:24.514ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
11992024-05-15T03:47:24.514ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/network-bootstore-config", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "f5291f0c-fa53-4645-a7aa-de273924bea9", "content-length": "467", "date": "Wed, 15 May 2024 03:47:24 GMT"} })
12002024-05-15T03:47:24.514ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
12012024-05-15T03:47:24.514ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
12022024-05-15T03:47:24.517ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
12032024-05-15T03:47:24.517ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
12042024-05-15T03:47:24.520ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
12052024-05-15T03:47:24.520ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
12062024-05-15T03:47:24.521ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
12072024-05-15T03:47:24.521ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
12082024-05-15T03:47:24.527ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 314.60216ms iteration = 12
12092024-05-15T03:47:24.807ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 146 local_addr = [fd00:1122:3344:101::c]:55127 method = GET remote_addr = [fd00:1122:3344:101::d]:37120 req_id = b6bfa519-a52c-4427-ab93-9766e0691e2f response_code = 200 uri = //86c56a1d-af69-43b6-8206-e5d1baefadb2
12102024-05-15T03:47:34.212ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 35 reason = Timeout
12112024-05-15T03:47:34.212ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12122024-05-15T03:47:34.212ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12132024-05-15T03:47:34.214ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12142024-05-15T03:47:34.216ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
12152024-05-15T03:47:34.216ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12162024-05-15T03:47:34.216ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12172024-05-15T03:47:34.218ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12182024-05-15T03:47:34.218ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12192024-05-15T03:47:34.219ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12202024-05-15T03:47:34.222ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0b55a4be-1fdc-4b91-a3a7-f964eab4522f, lookup_type: ById(0b55a4be-1fdc-4b91-a3a7-f964eab4522f) } result = Ok(())
12212024-05-15T03:47:34.222ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12222024-05-15T03:47:34.222ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12232024-05-15T03:47:34.228ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f original_time_created = 2024-05-15 03:41:43.607350 UTC target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f time_created = 2024-05-15 03:41:43.607350 UTC
12242024-05-15T03:47:34.228ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 16.431087ms iteration = 35
12252024-05-15T03:47:34.807ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 232 local_addr = [fd00:1122:3344:101::c]:55127 method = GET remote_addr = [fd00:1122:3344:101::d]:37120 req_id = 0fb53f08-52c0-44db-baf5-9517dec7b075 response_code = 200 uri = //86c56a1d-af69-43b6-8206-e5d1baefadb2
12262024-05-15T03:47:44.213ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 36 reason = Timeout
12272024-05-15T03:47:44.213ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12282024-05-15T03:47:44.213ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12292024-05-15T03:47:44.215ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12302024-05-15T03:47:44.217ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
12312024-05-15T03:47:44.217ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12322024-05-15T03:47:44.217ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12332024-05-15T03:47:44.219ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12342024-05-15T03:47:44.219ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12352024-05-15T03:47:44.220ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12362024-05-15T03:47:44.223ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0b55a4be-1fdc-4b91-a3a7-f964eab4522f, lookup_type: ById(0b55a4be-1fdc-4b91-a3a7-f964eab4522f) } result = Ok(())
12372024-05-15T03:47:44.223ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12382024-05-15T03:47:44.223ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12392024-05-15T03:47:44.229ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f original_time_created = 2024-05-15 03:41:43.607350 UTC target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f time_created = 2024-05-15 03:41:43.607350 UTC
12402024-05-15T03:47:44.229ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 16.132844ms iteration = 36
12412024-05-15T03:47:44.808ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 162 local_addr = [fd00:1122:3344:101::c]:55127 method = GET remote_addr = [fd00:1122:3344:101::d]:37120 req_id = f931a9c3-aa72-413d-8fd5-a6c3b5701a72 response_code = 200 uri = //86c56a1d-af69-43b6-8206-e5d1baefadb2
12422024-05-15T03:47:54.212ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 37 reason = Timeout
12432024-05-15T03:47:54.212ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 7 reason = Timeout
12442024-05-15T03:47:54.212ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
12452024-05-15T03:47:54.212ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
12462024-05-15T03:47:54.212ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
12472024-05-15T03:47:54.212ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 9 reason = Timeout
12482024-05-15T03:47:54.212ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 13 reason = Timeout
12492024-05-15T03:47:54.212ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
12502024-05-15T03:47:54.212ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 13 reason = Timeout
12512024-05-15T03:47:54.212ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
12522024-05-15T03:47:54.212ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 13 reason = Timeout
12532024-05-15T03:47:54.212ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:46
12542024-05-15T03:47:54.212ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 7 reason = Timeout
12552024-05-15T03:47:54.212ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 9 reason = Timeout
12562024-05-15T03:47:54.212ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 7 reason = Timeout
12572024-05-15T03:47:54.212ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
12582024-05-15T03:47:54.212ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 8 reason = Timeout
12592024-05-15T03:47:54.212ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 7 reason = Timeout
12602024-05-15T03:47:54.213ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/blueprint_execution.rs:65 reason = blueprint disabled target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f
12612024-05-15T03:47:54.213ZWARNnexus (BackgroundTasks): Metric producer GC: statically disabled pending omicron#5284 background_task = metrics_producer_gc file = nexus/src/app/background/metrics_producer_gc.rs:43
12622024-05-15T03:47:54.213ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 17.481\u{b5}s iteration = 8
12632024-05-15T03:47:54.213ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 9.92\u{b5}s iteration = 7
12642024-05-15T03:47:54.213ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 7 reason = Timeout
12652024-05-15T03:47:54.213ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 7 reason = Timeout
12662024-05-15T03:47:54.213ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
12672024-05-15T03:47:54.213ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
12682024-05-15T03:47:54.213ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
12692024-05-15T03:47:54.213ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
12702024-05-15T03:47:54.213ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
12712024-05-15T03:47:54.213ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
12722024-05-15T03:47:54.213ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
12732024-05-15T03:47:54.213ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
12742024-05-15T03:47:54.213ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
12752024-05-15T03:47:54.213ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
12762024-05-15T03:47:54.213ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
12772024-05-15T03:47:54.214ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
12782024-05-15T03:47:54.214ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 13 reason = Timeout
12792024-05-15T03:47:54.214ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 13 reason = Timeout
12802024-05-15T03:47:54.214ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
12812024-05-15T03:47:54.214ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
12822024-05-15T03:47:54.214ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 13 reason = Timeout
12832024-05-15T03:47:54.214ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
12842024-05-15T03:47:54.214ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 13 reason = Timeout
12852024-05-15T03:47:54.214ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:44
12862024-05-15T03:47:54.214ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:48
12872024-05-15T03:47:54.214ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 6.78\u{b5}s iteration = 13
12882024-05-15T03:47:54.214ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
12892024-05-15T03:47:54.214ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
12902024-05-15T03:47:54.214ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 13 reason = Timeout
12912024-05-15T03:47:54.214ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
12922024-05-15T03:47:54.214ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
12932024-05-15T03:47:54.214ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("3a1867b9-e5ea-4194-bfc7-0db4fb07d69e.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("9321e01e-813d-4e4f-8668-60649e9711dc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("9321e01e-813d-4e4f-8668-60649e9711dc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::8)) }], valid_until: Instant { tv_sec: 3851, tv_nsec: 639950694 } })
12942024-05-15T03:47:54.214ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("2b87d77b-ee3f-43c8-bced-dd105bbcdaa3.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("510fd65b-f60d-4faa-85be-82d26f4dd427.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("65e0e735-7f1e-479a-8127-a80c8289fcdb.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("65e0e735-7f1e-479a-8127-a80c8289fcdb.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:1::1)) }], valid_until: Instant { tv_sec: 3851, tv_nsec: 639961875 } })
12952024-05-15T03:47:54.215ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
12962024-05-15T03:47:54.215ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12972024-05-15T03:47:54.215ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12982024-05-15T03:47:54.216ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12992024-05-15T03:47:54.216ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:92
13002024-05-15T03:47:54.216ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.557796ms iteration = 13
13012024-05-15T03:47:54.216ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13022024-05-15T03:47:54.217ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13032024-05-15T03:47:54.217ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
13042024-05-15T03:47:54.217ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 4.760568ms iteration = 7
13052024-05-15T03:47:54.218ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
13062024-05-15T03:47:54.218ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 5.84254ms iteration = 7
13072024-05-15T03:47:54.220ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
13082024-05-15T03:47:54.220ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
13092024-05-15T03:47:54.220ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
13102024-05-15T03:47:54.220ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
13112024-05-15T03:47:54.220ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
13122024-05-15T03:47:54.221ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
13132024-05-15T03:47:54.222ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
13142024-05-15T03:47:54.222ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
13152024-05-15T03:47:54.222ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
13162024-05-15T03:47:54.222ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
13172024-05-15T03:47:54.222ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13182024-05-15T03:47:54.222ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
13192024-05-15T03:47:54.222ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
13202024-05-15T03:47:54.222ZWARNnexus (BackgroundTasks): physical disk adoption task started background_task = physical_disk_adoption file = nexus/src/app/background/physical_disk_adoption.rs:100
13212024-05-15T03:47:54.222ZWARNnexus (BackgroundTasks): Physical Disk Adoption: skipped background_task = physical_disk_adoption file = nexus/src/app/background/physical_disk_adoption.rs:104 reason = no inventory
13222024-05-15T03:47:54.222ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 8.78085ms iteration = 13
13232024-05-15T03:47:54.223ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
13242024-05-15T03:47:54.223ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3851, tv_nsec: 648538982 } })
13252024-05-15T03:47:54.223ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
13262024-05-15T03:47:54.223ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
13272024-05-15T03:47:54.223ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
13282024-05-15T03:47:54.223ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
13292024-05-15T03:47:54.223ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13302024-05-15T03:47:54.223ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
13312024-05-15T03:47:54.224ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
13322024-05-15T03:47:54.224ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
13332024-05-15T03:47:54.224ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
13342024-05-15T03:47:54.224ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
13352024-05-15T03:47:54.224ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13362024-05-15T03:47:54.224ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
13372024-05-15T03:47:54.224ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13382024-05-15T03:47:54.224ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
13392024-05-15T03:47:54.225ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
13402024-05-15T03:47:54.225ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13412024-05-15T03:47:54.225ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3851, tv_nsec: 650773865 } })
13422024-05-15T03:47:54.225ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
13432024-05-15T03:47:54.226ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
13442024-05-15T03:47:54.226ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
13452024-05-15T03:47:54.226ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13462024-05-15T03:47:54.226ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13472024-05-15T03:47:54.229ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
13482024-05-15T03:47:54.229ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
13492024-05-15T03:47:54.229ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
13502024-05-15T03:47:54.230ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
13512024-05-15T03:47:54.230ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
13522024-05-15T03:47:54.231ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
13532024-05-15T03:47:54.231ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1022
13542024-05-15T03:47:54.231ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/instance_watcher.rs:469 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0
13552024-05-15T03:47:54.231ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 16.998363ms iteration = 13
13562024-05-15T03:47:54.231ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
13572024-05-15T03:47:54.231ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
13582024-05-15T03:47:54.232ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
13592024-05-15T03:47:54.232ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0b55a4be-1fdc-4b91-a3a7-f964eab4522f, lookup_type: ById(0b55a4be-1fdc-4b91-a3a7-f964eab4522f) } result = Ok(())
13602024-05-15T03:47:54.232ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
13612024-05-15T03:47:54.232ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
13622024-05-15T03:47:54.232ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
13632024-05-15T03:47:54.232ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
13642024-05-15T03:47:54.232ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
13652024-05-15T03:47:54.233ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1022
13662024-05-15T03:47:54.233ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
13672024-05-15T03:47:54.233ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
13682024-05-15T03:47:54.233ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
13692024-05-15T03:47:54.233ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
13702024-05-15T03:47:54.233ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
13712024-05-15T03:47:54.234ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
13722024-05-15T03:47:54.234ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
13732024-05-15T03:47:54.234ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3851, tv_nsec: 659776697 } })
13742024-05-15T03:47:54.234ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13752024-05-15T03:47:54.235ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
13762024-05-15T03:47:54.235ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 3698e770-e01a-43e5-856e-b62ced4fb671 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
13772024-05-15T03:47:54.235ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
13782024-05-15T03:47:54.236ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
13792024-05-15T03:47:54.236ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1022 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4
13802024-05-15T03:47:54.237ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13812024-05-15T03:47:54.238ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
13822024-05-15T03:47:54.238ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
13832024-05-15T03:47:54.238ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
13842024-05-15T03:47:54.240ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f original_time_created = 2024-05-15 03:41:43.607350 UTC target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f time_created = 2024-05-15 03:41:43.607350 UTC
13852024-05-15T03:47:54.240ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 27.857844ms iteration = 37
13862024-05-15T03:47:54.242ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
13872024-05-15T03:47:54.242ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
13882024-05-15T03:47:54.242ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
13892024-05-15T03:47:54.257ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 3698e770-e01a-43e5-856e-b62ced4fb671 dns_zone_name = control-plane.oxide.internal found_so_far = 49 version = 1
13902024-05-15T03:47:54.257ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
13912024-05-15T03:47:54.257ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2024-05-15 03:41:56.187733 UTC dns_group = internal generation = 1
13922024-05-15T03:47:54.257ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 44.927028ms iteration = 7
13932024-05-15T03:47:54.258ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
13942024-05-15T03:47:54.259ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
13952024-05-15T03:47:54.260ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
13962024-05-15T03:47:54.261ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
13972024-05-15T03:47:54.262ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13982024-05-15T03:47:54.265ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 10925274-3dbb-4558-9fc7-9ec149150c20, lookup_type: ById(10925274-3dbb-4558-9fc7-9ec149150c20) } result = Ok(())
13992024-05-15T03:47:54.265ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
14002024-05-15T03:47:54.265ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
14012024-05-15T03:47:54.267ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 52.851039ms iteration = 13
14022024-05-15T03:47:54.273ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
14032024-05-15T03:47:54.273ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 52cc15d1-4ce1-4e38-8c6c-f9f099a8a5be dns_zone_name = oxide.test found_so_far = 0 version = 2
14042024-05-15T03:47:54.273ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
14052024-05-15T03:47:54.274ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
14062024-05-15T03:47:54.275ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14072024-05-15T03:47:54.277ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
14082024-05-15T03:47:54.277ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
14092024-05-15T03:47:54.277ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
14102024-05-15T03:47:54.279ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 52cc15d1-4ce1-4e38-8c6c-f9f099a8a5be dns_zone_name = oxide.test found_so_far = 1 version = 2
14112024-05-15T03:47:54.279ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
14122024-05-15T03:47:54.279ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2024-05-15 03:41:57.099854 UTC dns_group = external generation = 2
14132024-05-15T03:47:54.279ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 66.810781ms iteration = 7
14142024-05-15T03:47:54.286ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2024-05-15T03:41:54.396604Z, time_modified: 2024-05-15T03:41:54.396604Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 39f7e4ba-533e-4258-9f79-1873f513f101, db_silo: Silo { identity: SiloIdentity { id: 39f7e4ba-533e-4258-9f79-1873f513f101, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-05-15T03:41:57.052187Z, time_modified: 2024-05-15T03:41:57.052187Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "370e7f12febb2dc0abbebc4192727b23c734c770fdab3368ed3fd759491f6fed" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 39f7e4ba-533e-4258-9f79-1873f513f101, db_silo: Silo { identity: SiloIdentity { id: 39f7e4ba-533e-4258-9f79-1873f513f101, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-05-15T03:41:57.052187Z, time_modified: 2024-05-15T03:41:57.052187Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(3)) }, tls_certs: [TlsCertificate { digest: "370e7f12febb2dc0abbebc4192727b23c734c770fdab3368ed3fd759491f6fed" }] }) }
14152024-05-15T03:47:54.286ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 73.391449ms iteration = 7
14162024-05-15T03:47:54.355ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
14172024-05-15T03:47:54.388ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1030 zone_address = fd00:1122:3344:101::2
14182024-05-15T03:47:54.388ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
14192024-05-15T03:47:54.389ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "531608e2-b2ba-4cb3-8efd-d45853c3769f", "content-length": "26", "date": "Wed, 15 May 2024 03:47:54 GMT"} })
14202024-05-15T03:47:54.389ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1035 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
14212024-05-15T03:47:54.390ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1067 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
14222024-05-15T03:47:54.395ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1030 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 zone_address = fd00:1122:3344:101::2
14232024-05-15T03:47:54.395ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
14242024-05-15T03:47:54.396ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "39197d87-7610-4014-a96a-de4a26b62e14", "content-length": "26", "date": "Wed, 15 May 2024 03:47:54 GMT"} })
14252024-05-15T03:47:54.396ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1035 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
14262024-05-15T03:47:54.397ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1067 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4
14272024-05-15T03:47:54.464ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
14282024-05-15T03:47:54.506ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1030 zone_address = fd00:1122:3344:101::2
14292024-05-15T03:47:54.506ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
14302024-05-15T03:47:54.506ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "d36f3036-3304-4bf3-b217-31857a5734c3", "content-length": "26", "date": "Wed, 15 May 2024 03:47:54 GMT"} })
14312024-05-15T03:47:54.506ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1035 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
14322024-05-15T03:47:54.508ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1067 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
14332024-05-15T03:47:54.514ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
14342024-05-15T03:47:54.514ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::8)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "6969809b-abb8-4d1b-a151-55ed71a129f9", "date": "Wed, 15 May 2024 03:47:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
14352024-05-15T03:47:54.514ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
14362024-05-15T03:47:54.516ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::9)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "d56eb90c-39b3-4478-b279-547c1d8310ca", "date": "Wed, 15 May 2024 03:47:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
14372024-05-15T03:47:54.516ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
14382024-05-15T03:47:54.517ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
14392024-05-15T03:47:54.517ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 304.86643ms iteration = 9
14402024-05-15T03:47:54.538ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 326.154157ms iteration = 13
14412024-05-15T03:47:54.655ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
14422024-05-15T03:47:54.716ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14432024-05-15T03:47:54.717ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14442024-05-15T03:47:54.719ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:144 port = SwitchPort { id: 89eb00e0-b12a-4307-aac5-625fd364b560, rack_id: 76c05259-5403-45cc-a1c6-20f84d7e54b4, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(7f8c724a-b675-4fd5-8a78-6b7ffc7a3247) } rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0
14452024-05-15T03:47:54.719ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14462024-05-15T03:47:54.719ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14472024-05-15T03:47:54.721ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
14482024-05-15T03:47:54.721ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "c8b0a396-2e9b-4c04-a3f9-3ceb10c8c85f", "content-length": "2", "date": "Wed, 15 May 2024 03:47:54 GMT"} })
14492024-05-15T03:47:54.723ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
14502024-05-15T03:47:54.724ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
14512024-05-15T03:47:54.725ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 512.504978ms iteration = 13
14522024-05-15T03:47:54.727ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
14532024-05-15T03:47:54.727ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/port/qsfp0/settings", query: Some("tag=omicron"), fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "998c1a35-11b2-4d88-8699-d55a5237a318", "content-length": "121", "date": "Wed, 15 May 2024 03:47:54 GMT"} })
14542024-05-15T03:47:54.727ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1643 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: None, kr: false, lane: Some(LinkId(0)), speed: Speed40G } }} } switch_port_id = "qsfp0"
14552024-05-15T03:47:54.727ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1670 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 7f8c724a-b675-4fd5-8a78-6b7ffc7a3247
14562024-05-15T03:47:54.727ZERROnexus (BackgroundTasks): unable to retrieve routes from switch background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1782 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0
14572024-05-15T03:47:54.727ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:369 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 routes = {}
14582024-05-15T03:47:54.727ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:373 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None)}}
14592024-05-15T03:47:54.727ZWARNnexus (BackgroundTasks): no discovered routes from switch. it is possible that an earlier api call failed. background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1511 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0
14602024-05-15T03:47:54.727ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:382 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 routes = {}
14612024-05-15T03:47:54.727ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:388 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 routes = {}
14622024-05-15T03:47:54.727ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:408 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4
14632024-05-15T03:47:54.727ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14642024-05-15T03:47:54.728ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14652024-05-15T03:47:54.734ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
14662024-05-15T03:47:54.735ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv4", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "9dfa8d2a-f569-498b-90da-c68f7770aea3", "content-length": "2", "date": "Wed, 15 May 2024 03:47:54 GMT"} })
14672024-05-15T03:47:54.735ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
14682024-05-15T03:47:54.735ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv6", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "0a0fddbf-fa9e-4d18-983b-d6a67fbf4e2b", "content-length": "52", "date": "Wed, 15 May 2024 03:47:54 GMT"} })
14692024-05-15T03:47:54.735ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [V4(Ipv4Network { addr: 10.151.1.174, prefix: 24 })], port: "qsfp0" }] file = nexus/src/app/background/sync_switch_configuration.rs:457 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0
14702024-05-15T03:47:54.735ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
14712024-05-15T03:47:54.740ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
14722024-05-15T03:47:54.742ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:1::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "6756cff4-3ada-4526-8a09-4745abc62de2", "date": "Wed, 15 May 2024 03:47:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
14732024-05-15T03:47:54.742ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
14742024-05-15T03:47:54.743ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:2::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "f599d986-a8a7-45f8-8d3c-1ef78517e857", "date": "Wed, 15 May 2024 03:47:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
14752024-05-15T03:47:54.743ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
14762024-05-15T03:47:54.744ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:3::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "2312fa25-9286-496b-a761-f66909372158", "date": "Wed, 15 May 2024 03:47:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
14772024-05-15T03:47:54.744ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
14782024-05-15T03:47:54.746ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
14792024-05-15T03:47:54.746ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 533.603914ms iteration = 9
14802024-05-15T03:47:54.785ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/switch-ports", query: None, fragment: None }, status: 204, headers: {"x-request-id": "4600a3f4-0af1-4371-bb89-31f9d078f0af", "date": "Wed, 15 May 2024 03:47:54 GMT"} })
14812024-05-15T03:47:54.785ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:759 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch = Switch0
14822024-05-15T03:47:54.786ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
14832024-05-15T03:47:54.786ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/network-bootstore-config", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "c3d6bbbf-2986-442f-b730-bbeedcdeb5ed", "content-length": "467", "date": "Wed, 15 May 2024 03:47:54 GMT"} })
14842024-05-15T03:47:54.786ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14852024-05-15T03:47:54.787ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14862024-05-15T03:47:54.789ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14872024-05-15T03:47:54.789ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14882024-05-15T03:47:54.792ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14892024-05-15T03:47:54.792ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14902024-05-15T03:47:54.793ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
14912024-05-15T03:47:54.793ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
14922024-05-15T03:47:54.799ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 585.851636ms iteration = 13
14932024-05-15T03:47:54.808ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 823 local_addr = [fd00:1122:3344:101::c]:55127 method = GET remote_addr = [fd00:1122:3344:101::d]:37120 req_id = 02016e09-543c-43c2-b5bb-a6a169f5aeeb response_code = 200 uri = //86c56a1d-af69-43b6-8206-e5d1baefadb2
14942024-05-15T03:47:54.811ZDEBGnexus (producer-registration-task): registering / renewing oximeter producer lease with Nexus
14952024-05-15T03:47:54.811ZDEBGnexus (producer-registration-task): using nexus address for registration addr = [fd00:1122:3344:101::c]:12221
14962024-05-15T03:47:54.841ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::c]:12221/metrics/producers
14972024-05-15T03:47:54.841ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::c]:12221 remote_addr = [fd00:1122:3344:101::c]:54128
14982024-05-15T03:47:54.841ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::c]:54128 req_id = 2bb76948-5a12-4b78-acaa-be16bacb35cf roles = RoleSet { roles: {} } uri = /metrics/producers
14992024-05-15T03:47:54.841ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::c]:54128 req_id = 2bb76948-5a12-4b78-acaa-be16bacb35cf resource = Database result = Ok(()) uri = /metrics/producers
15002024-05-15T03:47:54.872ZINFOnexus: registered oximeter collector client file = nexus/src/app/oximeter.rs:338 id = f12aa7e2-1873-452c-b215-b2593f228861
15012024-05-15T03:47:54.872ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::c]:54128 req_id = 2bb76948-5a12-4b78-acaa-be16bacb35cf roles = RoleSet { roles: {} } uri = /metrics/producers
15022024-05-15T03:47:54.872ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::c]:54128 req_id = 2bb76948-5a12-4b78-acaa-be16bacb35cf resource = Database result = Ok(()) uri = /metrics/producers
15032024-05-15T03:47:54.877ZDEBGnexus: client request body = Some(Body) method = POST oximeter-collector = f12aa7e2-1873-452c-b215-b2593f228861 uri = http://[fd00:1122:3344:101::d]:12223/producers
15042024-05-15T03:47:54.878ZDEBGnexus: client response oximeter-collector = f12aa7e2-1873-452c-b215-b2593f228861 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::d)), port: Some(12223), path: "/producers", query: None, fragment: None }, status: 204, headers: {"x-request-id": "1f888b88-5d3d-4dd1-a161-c3e971873e38", "date": "Wed, 15 May 2024 03:47:54 GMT"} })
15052024-05-15T03:47:54.878ZINFOnexus: assigned collector to new producer collector_id = f12aa7e2-1873-452c-b215-b2593f228861 file = nexus/src/app/oximeter.rs:125 producer_id = 86c56a1d-af69-43b6-8206-e5d1baefadb2
15062024-05-15T03:47:54.879ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 38472 local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::c]:54128 req_id = 2bb76948-5a12-4b78-acaa-be16bacb35cf response_code = 201 uri = /metrics/producers
15072024-05-15T03:47:54.879ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::c)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, status: 201, headers: {"content-type": "application/json", "x-request-id": "2bb76948-5a12-4b78-acaa-be16bacb35cf", "content-length": "41", "date": "Wed, 15 May 2024 03:47:54 GMT"} })
15082024-05-15T03:47:54.881ZDEBGnexus (producer-registration-task): registered with nexus successfully
15092024-05-15T03:47:54.881ZDEBGnexus (producer-registration-task): pausing until time to renew lease lease_duration = 600s wait_period = 150s
15102024-05-15T03:48:04.213ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 38 reason = Timeout
15112024-05-15T03:48:04.213ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15122024-05-15T03:48:04.214ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15132024-05-15T03:48:04.215ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15142024-05-15T03:48:04.218ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
15152024-05-15T03:48:04.218ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15162024-05-15T03:48:04.218ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15172024-05-15T03:48:04.220ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15182024-05-15T03:48:04.220ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15192024-05-15T03:48:04.221ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15202024-05-15T03:48:04.224ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0b55a4be-1fdc-4b91-a3a7-f964eab4522f, lookup_type: ById(0b55a4be-1fdc-4b91-a3a7-f964eab4522f) } result = Ok(())
15212024-05-15T03:48:04.224ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15222024-05-15T03:48:04.224ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15232024-05-15T03:48:04.230ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f original_time_created = 2024-05-15 03:41:43.607350 UTC target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f time_created = 2024-05-15 03:41:43.607350 UTC
15242024-05-15T03:48:04.230ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 16.750501ms iteration = 38
15252024-05-15T03:48:04.880ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 142 local_addr = [fd00:1122:3344:101::c]:55127 method = GET remote_addr = [fd00:1122:3344:101::d]:37120 req_id = 921b16ab-dc0b-4e62-b14e-031c2b3f1606 response_code = 200 uri = //86c56a1d-af69-43b6-8206-e5d1baefadb2
15262024-05-15T03:48:14.213ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 39 reason = Timeout
15272024-05-15T03:48:14.213ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15282024-05-15T03:48:14.213ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15292024-05-15T03:48:14.214ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15302024-05-15T03:48:14.217ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
15312024-05-15T03:48:14.217ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15322024-05-15T03:48:14.217ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15332024-05-15T03:48:14.219ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15342024-05-15T03:48:14.219ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15352024-05-15T03:48:14.221ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15362024-05-15T03:48:14.223ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0b55a4be-1fdc-4b91-a3a7-f964eab4522f, lookup_type: ById(0b55a4be-1fdc-4b91-a3a7-f964eab4522f) } result = Ok(())
15372024-05-15T03:48:14.223ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15382024-05-15T03:48:14.224ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15392024-05-15T03:48:14.230ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f original_time_created = 2024-05-15 03:41:43.607350 UTC target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f time_created = 2024-05-15 03:41:43.607350 UTC
15402024-05-15T03:48:14.230ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 17.108685ms iteration = 39
15412024-05-15T03:48:14.880ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 144 local_addr = [fd00:1122:3344:101::c]:55127 method = GET remote_addr = [fd00:1122:3344:101::d]:37120 req_id = b123cdff-f78a-4dd2-a134-47b5749c1764 response_code = 200 uri = //86c56a1d-af69-43b6-8206-e5d1baefadb2
15422024-05-15T03:48:21.210ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::c]:12221 remote_addr = [fd00:1122:3344:101::1:0]:36144
15432024-05-15T03:48:21.210ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::1:0]:36144 req_id = 2864d153-8335-424c-8596-024ae1f33832 roles = RoleSet { roles: {} } uri = /metrics/producers
15442024-05-15T03:48:21.211ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::1:0]:36144 req_id = 2864d153-8335-424c-8596-024ae1f33832 resource = Database result = Ok(()) uri = /metrics/producers
15452024-05-15T03:48:21.243ZINFOnexus: registered oximeter collector client file = nexus/src/app/oximeter.rs:338 id = f12aa7e2-1873-452c-b215-b2593f228861
15462024-05-15T03:48:21.243ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::1:0]:36144 req_id = 2864d153-8335-424c-8596-024ae1f33832 roles = RoleSet { roles: {} } uri = /metrics/producers
15472024-05-15T03:48:21.244ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::1:0]:36144 req_id = 2864d153-8335-424c-8596-024ae1f33832 resource = Database result = Ok(()) uri = /metrics/producers
15482024-05-15T03:48:21.250ZDEBGnexus: client request body = Some(Body) method = POST oximeter-collector = f12aa7e2-1873-452c-b215-b2593f228861 uri = http://[fd00:1122:3344:101::d]:12223/producers
15492024-05-15T03:48:21.250ZDEBGnexus: client response oximeter-collector = f12aa7e2-1873-452c-b215-b2593f228861 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::d)), port: Some(12223), path: "/producers", query: None, fragment: None }, status: 204, headers: {"x-request-id": "ef669b92-93bd-48e6-814b-72818ac60273", "date": "Wed, 15 May 2024 03:48:21 GMT"} })
15502024-05-15T03:48:21.250ZINFOnexus: assigned collector to new producer collector_id = f12aa7e2-1873-452c-b215-b2593f228861 file = nexus/src/app/oximeter.rs:125 producer_id = 440eb2df-a50f-4b4f-b440-652023bc4285
15512024-05-15T03:48:21.251ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 40832 local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::1:0]:36144 req_id = 2864d153-8335-424c-8596-024ae1f33832 response_code = 201 uri = /metrics/producers
15522024-05-15T03:48:24.213ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 40 reason = Timeout
15532024-05-15T03:48:24.213ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 14 reason = Timeout
15542024-05-15T03:48:24.213ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
15552024-05-15T03:48:24.213ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 14 reason = Timeout
15562024-05-15T03:48:24.213ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:46
15572024-05-15T03:48:24.213ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
15582024-05-15T03:48:24.213ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 14 reason = Timeout
15592024-05-15T03:48:24.213ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
15602024-05-15T03:48:24.213ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
15612024-05-15T03:48:24.213ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
15622024-05-15T03:48:24.213ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
15632024-05-15T03:48:24.214ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 14 reason = Timeout
15642024-05-15T03:48:24.214ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
15652024-05-15T03:48:24.214ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 14 reason = Timeout
15662024-05-15T03:48:24.214ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 14 reason = Timeout
15672024-05-15T03:48:24.214ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:44
15682024-05-15T03:48:24.214ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:48
15692024-05-15T03:48:24.214ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
15702024-05-15T03:48:24.214ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 7.22\u{b5}s iteration = 14
15712024-05-15T03:48:24.214ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 14 reason = Timeout
15722024-05-15T03:48:24.214ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
15732024-05-15T03:48:24.214ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
15742024-05-15T03:48:24.214ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
15752024-05-15T03:48:24.214ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
15762024-05-15T03:48:24.215ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 14 reason = Timeout
15772024-05-15T03:48:24.215ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
15782024-05-15T03:48:24.215ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15792024-05-15T03:48:24.216ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
15802024-05-15T03:48:24.216ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15812024-05-15T03:48:24.216ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3881, tv_nsec: 641097586 } })
15822024-05-15T03:48:24.216ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15832024-05-15T03:48:24.216ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1022
15842024-05-15T03:48:24.216ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:92
15852024-05-15T03:48:24.216ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.719978ms iteration = 14
15862024-05-15T03:48:24.216ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15872024-05-15T03:48:24.217ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3881, tv_nsec: 641736923 } })
15882024-05-15T03:48:24.217ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15892024-05-15T03:48:24.219ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1022
15902024-05-15T03:48:24.221ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
15912024-05-15T03:48:24.221ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
15922024-05-15T03:48:24.221ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
15932024-05-15T03:48:24.221ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
15942024-05-15T03:48:24.221ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
15952024-05-15T03:48:24.221ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
15962024-05-15T03:48:24.222ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
15972024-05-15T03:48:24.222ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
15982024-05-15T03:48:24.222ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
15992024-05-15T03:48:24.222ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16002024-05-15T03:48:24.222ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
16012024-05-15T03:48:24.222ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
16022024-05-15T03:48:24.222ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
16032024-05-15T03:48:24.222ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
16042024-05-15T03:48:24.222ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
16052024-05-15T03:48:24.230ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
16062024-05-15T03:48:24.230ZWARNnexus (BackgroundTasks): physical disk adoption task started background_task = physical_disk_adoption file = nexus/src/app/background/physical_disk_adoption.rs:100
16072024-05-15T03:48:24.230ZWARNnexus (BackgroundTasks): Physical Disk Adoption: skipped background_task = physical_disk_adoption file = nexus/src/app/background/physical_disk_adoption.rs:104 reason = no inventory
16082024-05-15T03:48:24.230ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 15.964212ms iteration = 14
16092024-05-15T03:48:24.230ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
16102024-05-15T03:48:24.235ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16112024-05-15T03:48:24.235ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
16122024-05-15T03:48:24.239ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3881, tv_nsec: 663692297 } })
16132024-05-15T03:48:24.239ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16142024-05-15T03:48:24.239ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1022 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4
16152024-05-15T03:48:24.242ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0b55a4be-1fdc-4b91-a3a7-f964eab4522f, lookup_type: ById(0b55a4be-1fdc-4b91-a3a7-f964eab4522f) } result = Ok(())
16162024-05-15T03:48:24.242ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
16172024-05-15T03:48:24.243ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
16182024-05-15T03:48:24.250ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f original_time_created = 2024-05-15 03:41:43.607350 UTC target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f time_created = 2024-05-15 03:41:43.607350 UTC
16192024-05-15T03:48:24.250ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 36.962258ms iteration = 40
16202024-05-15T03:48:24.254ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
16212024-05-15T03:48:24.254ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
16222024-05-15T03:48:24.255ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
16232024-05-15T03:48:24.256ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
16242024-05-15T03:48:24.257ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16252024-05-15T03:48:24.260ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 10925274-3dbb-4558-9fc7-9ec149150c20, lookup_type: ById(10925274-3dbb-4558-9fc7-9ec149150c20) } result = Ok(())
16262024-05-15T03:48:24.260ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
16272024-05-15T03:48:24.260ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
16282024-05-15T03:48:24.261ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 47.611756ms iteration = 14
16292024-05-15T03:48:24.288ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1030 zone_address = fd00:1122:3344:101::2
16302024-05-15T03:48:24.288ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
16312024-05-15T03:48:24.288ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "f9ee7f73-0640-4308-9da3-c0257722653e", "content-length": "26", "date": "Wed, 15 May 2024 03:48:24 GMT"} })
16322024-05-15T03:48:24.288ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1035 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
16332024-05-15T03:48:24.290ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1067 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
16342024-05-15T03:48:24.293ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1030 zone_address = fd00:1122:3344:101::2
16352024-05-15T03:48:24.293ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
16362024-05-15T03:48:24.293ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "b4dfb010-fa5e-4c5b-b066-b66223e55824", "content-length": "26", "date": "Wed, 15 May 2024 03:48:24 GMT"} })
16372024-05-15T03:48:24.294ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1035 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
16382024-05-15T03:48:24.296ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1067 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
16392024-05-15T03:48:24.317ZDEBGnexus (BackgroundTasks): client request SledAgent = 10925274-3dbb-4558-9fc7-9ec149150c20 background_task = instance_watcher body = None method = GET uri = http://[fd00:1122:3344:101::1]:12345/instances/440eb2df-a50f-4b4f-b440-652023bc4285/state
16402024-05-15T03:48:24.318ZDEBGnexus (BackgroundTasks): client response SledAgent = 10925274-3dbb-4558-9fc7-9ec149150c20 background_task = instance_watcher result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/instances/440eb2df-a50f-4b4f-b440-652023bc4285/state", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "52accb15-04e2-4307-9682-b3c5246ad354", "content-length": "313", "date": "Wed, 15 May 2024 03:48:24 GMT"} })
16412024-05-15T03:48:24.325ZDEBGnexus (BackgroundTasks): updating instance state background_task = instance_watcher instance_id = 440eb2df-a50f-4b4f-b440-652023bc4285 state = Running
16422024-05-15T03:48:24.325ZINFOnexus (BackgroundTasks): received new runtime state from sled agent background_task = instance_watcher file = nexus/src/app/instance.rs:1979 instance_id = 440eb2df-a50f-4b4f-b440-652023bc4285 instance_state = InstanceRuntimeState { propolis_id: Some(e2113d0d-02dc-4c84-998e-a26c0d97d7d3), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-05-15T03:48:13.582282542Z } propolis_id = e2113d0d-02dc-4c84-998e-a26c0d97d7d3 vmm_state = VmmRuntimeState { state: Running, gen: Generation(3), time_updated: 2024-05-15T03:48:21.293903826Z }
16432024-05-15T03:48:24.325ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher instance_id = 440eb2df-a50f-4b4f-b440-652023bc4285 roles = RoleSet { roles: {} }
16442024-05-15T03:48:24.325ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher instance_id = 440eb2df-a50f-4b4f-b440-652023bc4285 resource = Database result = Ok(())
16452024-05-15T03:48:24.329ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher instance_id = 440eb2df-a50f-4b4f-b440-652023bc4285 roles = RoleSet { roles: {} }
16462024-05-15T03:48:24.329ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher instance_id = 440eb2df-a50f-4b4f-b440-652023bc4285 resource = Database result = Ok(())
16472024-05-15T03:48:24.331ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher instance_id = 440eb2df-a50f-4b4f-b440-652023bc4285 roles = RoleSet { roles: {} }
16482024-05-15T03:48:24.331ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher instance_id = 440eb2df-a50f-4b4f-b440-652023bc4285 resource = Database result = Ok(())
16492024-05-15T03:48:24.333ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher instance_id = 440eb2df-a50f-4b4f-b440-652023bc4285 roles = RoleSet { roles: {} }
16502024-05-15T03:48:24.333ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher instance_id = 440eb2df-a50f-4b4f-b440-652023bc4285 resource = Database result = Ok(())
16512024-05-15T03:48:24.334ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher instance_id = 440eb2df-a50f-4b4f-b440-652023bc4285 roles = RoleSet { roles: {} }
16522024-05-15T03:48:24.335ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher instance_id = 440eb2df-a50f-4b4f-b440-652023bc4285 resource = Database result = Ok(())
16532024-05-15T03:48:24.335ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1030 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 zone_address = fd00:1122:3344:101::2
16542024-05-15T03:48:24.335ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
16552024-05-15T03:48:24.335ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "d1c552ad-dc06-4677-a676-50b0eaf2f21b", "content-length": "26", "date": "Wed, 15 May 2024 03:48:24 GMT"} })
16562024-05-15T03:48:24.335ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1035 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
16572024-05-15T03:48:24.336ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher instance_id = 440eb2df-a50f-4b4f-b440-652023bc4285 roles = RoleSet { roles: {} }
16582024-05-15T03:48:24.336ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1067 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4
16592024-05-15T03:48:24.336ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher instance_id = 440eb2df-a50f-4b4f-b440-652023bc4285 resource = Database result = Ok(())
16602024-05-15T03:48:24.338ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher instance_id = 440eb2df-a50f-4b4f-b440-652023bc4285 roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16612024-05-15T03:48:24.345ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher instance_id = 440eb2df-a50f-4b4f-b440-652023bc4285 resource = Instance { parent: Project { parent: Silo { parent: Fleet, key: 39f7e4ba-533e-4258-9f79-1873f513f101, lookup_type: ById(39f7e4ba-533e-4258-9f79-1873f513f101) }, key: 12f5fa9b-b21b-473e-8969-bbdf20831093, lookup_type: ById(12f5fa9b-b21b-473e-8969-bbdf20831093) }, key: 440eb2df-a50f-4b4f-b440-652023bc4285, lookup_type: ById(440eb2df-a50f-4b4f-b440-652023bc4285) } result = Ok(())
16622024-05-15T03:48:24.347ZDEBGnexus (BackgroundTasks): instance state generation already advanced, won't touch network config background_task = instance_watcher instance_id = 440eb2df-a50f-4b4f-b440-652023bc4285
16632024-05-15T03:48:24.385ZINFOnexus (BackgroundTasks): instance and vmm updated by sled agent background_task = instance_watcher file = nexus/src/app/instance.rs:2112 instance_id = 440eb2df-a50f-4b4f-b440-652023bc4285 instance_updated = false propolis_id = e2113d0d-02dc-4c84-998e-a26c0d97d7d3 vmm_updated = false
16642024-05-15T03:48:24.385ZDEBGnexus (BackgroundTasks): update successful background_task = instance_watcher instance_id = 440eb2df-a50f-4b4f-b440-652023bc4285 instance_updated = false state = Running vmm_updated = false
16652024-05-15T03:48:24.389ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/instance_watcher.rs:469 pruned_instances = 0 total_completed = 1 total_failed = 0 total_incomplete = 0 total_instances = 1
16662024-05-15T03:48:24.389ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 173.54929ms iteration = 14
16672024-05-15T03:48:24.393ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
16682024-05-15T03:48:24.393ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "624b1346-259f-4064-a9f1-2cc5bd2e949c", "content-length": "2", "date": "Wed, 15 May 2024 03:48:23 GMT"} })
16692024-05-15T03:48:24.394ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
16702024-05-15T03:48:24.395ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
16712024-05-15T03:48:24.395ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 181.929665ms iteration = 14
16722024-05-15T03:48:24.396ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 183.281799ms iteration = 14
16732024-05-15T03:48:24.468ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
16742024-05-15T03:48:24.469ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
16752024-05-15T03:48:24.470ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:144 port = SwitchPort { id: 89eb00e0-b12a-4307-aac5-625fd364b560, rack_id: 76c05259-5403-45cc-a1c6-20f84d7e54b4, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(7f8c724a-b675-4fd5-8a78-6b7ffc7a3247) } rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0
16762024-05-15T03:48:24.470ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
16772024-05-15T03:48:24.470ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
16782024-05-15T03:48:24.478ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
16792024-05-15T03:48:24.478ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/port/qsfp0/settings", query: Some("tag=omicron"), fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "b6c5c4cd-53ce-42ca-bde6-8c3c7b745af0", "content-length": "121", "date": "Wed, 15 May 2024 03:48:23 GMT"} })
16802024-05-15T03:48:24.478ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1643 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: None, kr: false, lane: Some(LinkId(0)), speed: Speed40G } }} } switch_port_id = "qsfp0"
16812024-05-15T03:48:24.478ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1670 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 7f8c724a-b675-4fd5-8a78-6b7ffc7a3247
16822024-05-15T03:48:24.480ZERROnexus (BackgroundTasks): unable to retrieve routes from switch background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1782 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0
16832024-05-15T03:48:24.480ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:369 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 routes = {}
16842024-05-15T03:48:24.480ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:373 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None)}}
16852024-05-15T03:48:24.480ZWARNnexus (BackgroundTasks): no discovered routes from switch. it is possible that an earlier api call failed. background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1511 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0
16862024-05-15T03:48:24.480ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:382 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 routes = {}
16872024-05-15T03:48:24.480ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:388 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 routes = {}
16882024-05-15T03:48:24.480ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:408 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4
16892024-05-15T03:48:24.480ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
16902024-05-15T03:48:24.480ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
16912024-05-15T03:48:24.482ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
16922024-05-15T03:48:24.482ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv4", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "ab2d3718-92bb-4da7-b531-fe3c51072d89", "content-length": "2", "date": "Wed, 15 May 2024 03:48:23 GMT"} })
16932024-05-15T03:48:24.482ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
16942024-05-15T03:48:24.482ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv6", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "b729b60a-0c75-498a-af6d-bd47b5eee537", "content-length": "52", "date": "Wed, 15 May 2024 03:48:23 GMT"} })
16952024-05-15T03:48:24.482ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [V4(Ipv4Network { addr: 10.151.1.174, prefix: 24 })], port: "qsfp0" }] file = nexus/src/app/background/sync_switch_configuration.rs:457 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0
16962024-05-15T03:48:24.482ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
16972024-05-15T03:48:24.539ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/switch-ports", query: None, fragment: None }, status: 204, headers: {"x-request-id": "114887bd-1f76-42e5-af0d-2b56c5e951be", "date": "Wed, 15 May 2024 03:48:24 GMT"} })
16982024-05-15T03:48:24.539ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:759 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch = Switch0
16992024-05-15T03:48:24.539ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
17002024-05-15T03:48:24.540ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/network-bootstore-config", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "0b5e3bda-c189-4f14-b03c-af41b9cf4a97", "content-length": "467", "date": "Wed, 15 May 2024 03:48:24 GMT"} })
17012024-05-15T03:48:24.540ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
17022024-05-15T03:48:24.540ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
17032024-05-15T03:48:24.543ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
17042024-05-15T03:48:24.543ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
17052024-05-15T03:48:24.546ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
17062024-05-15T03:48:24.546ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
17072024-05-15T03:48:24.547ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
17082024-05-15T03:48:24.547ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
17092024-05-15T03:48:24.554ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 340.469873ms iteration = 14
17102024-05-15T03:48:24.882ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 2095 local_addr = [fd00:1122:3344:101::c]:55127 method = GET remote_addr = [fd00:1122:3344:101::d]:37120 req_id = 652a11b3-e9d1-46f4-87c5-1b410ea6484a response_code = 200 uri = //86c56a1d-af69-43b6-8206-e5d1baefadb2
17112024-05-15T03:48:34.213ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 41 reason = Timeout
17122024-05-15T03:48:34.213ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
17132024-05-15T03:48:34.214ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17142024-05-15T03:48:34.215ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17152024-05-15T03:48:34.217ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
17162024-05-15T03:48:34.217ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
17172024-05-15T03:48:34.218ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17182024-05-15T03:48:34.219ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
17192024-05-15T03:48:34.219ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17202024-05-15T03:48:34.220ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17212024-05-15T03:48:34.223ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0b55a4be-1fdc-4b91-a3a7-f964eab4522f, lookup_type: ById(0b55a4be-1fdc-4b91-a3a7-f964eab4522f) } result = Ok(())
17222024-05-15T03:48:34.223ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
17232024-05-15T03:48:34.223ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17242024-05-15T03:48:34.229ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f original_time_created = 2024-05-15 03:41:43.607350 UTC target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f time_created = 2024-05-15 03:41:43.607350 UTC
17252024-05-15T03:48:34.229ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 15.742551ms iteration = 41
17262024-05-15T03:48:34.881ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 165 local_addr = [fd00:1122:3344:101::c]:55127 method = GET remote_addr = [fd00:1122:3344:101::d]:37120 req_id = e1642cea-bf1b-48d9-90dc-3fc214a53785 response_code = 200 uri = //86c56a1d-af69-43b6-8206-e5d1baefadb2
17272024-05-15T03:48:44.213ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 42 reason = Timeout
17282024-05-15T03:48:44.213ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
17292024-05-15T03:48:44.213ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17302024-05-15T03:48:44.215ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17312024-05-15T03:48:44.218ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
17322024-05-15T03:48:44.218ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
17332024-05-15T03:48:44.218ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17342024-05-15T03:48:44.220ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
17352024-05-15T03:48:44.220ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17362024-05-15T03:48:44.221ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17372024-05-15T03:48:44.224ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0b55a4be-1fdc-4b91-a3a7-f964eab4522f, lookup_type: ById(0b55a4be-1fdc-4b91-a3a7-f964eab4522f) } result = Ok(())
17382024-05-15T03:48:44.224ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
17392024-05-15T03:48:44.224ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17402024-05-15T03:48:44.230ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f original_time_created = 2024-05-15 03:41:43.607350 UTC target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f time_created = 2024-05-15 03:41:43.607350 UTC
17412024-05-15T03:48:44.230ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 17.173635ms iteration = 42
17422024-05-15T03:48:44.881ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 229 local_addr = [fd00:1122:3344:101::c]:55127 method = GET remote_addr = [fd00:1122:3344:101::d]:37120 req_id = 21f289eb-20e7-4a33-b407-3296629cbde9 response_code = 200 uri = //86c56a1d-af69-43b6-8206-e5d1baefadb2
17432024-05-15T03:48:54.213ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 43 reason = Timeout
17442024-05-15T03:48:54.214ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 8 reason = Timeout
17452024-05-15T03:48:54.214ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
17462024-05-15T03:48:54.214ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
17472024-05-15T03:48:54.214ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
17482024-05-15T03:48:54.214ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 10 reason = Timeout
17492024-05-15T03:48:54.214ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 15 reason = Timeout
17502024-05-15T03:48:54.214ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
17512024-05-15T03:48:54.214ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 15 reason = Timeout
17522024-05-15T03:48:54.214ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:46
17532024-05-15T03:48:54.214ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 8 reason = Timeout
17542024-05-15T03:48:54.214ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 8 reason = Timeout
17552024-05-15T03:48:54.214ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 8 reason = Timeout
17562024-05-15T03:48:54.214ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
17572024-05-15T03:48:54.214ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
17582024-05-15T03:48:54.214ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 9 reason = Timeout
17592024-05-15T03:48:54.214ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/blueprint_execution.rs:65 reason = blueprint disabled target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f
17602024-05-15T03:48:54.214ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 14.601\u{b5}s iteration = 9
17612024-05-15T03:48:54.214ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 8 reason = Timeout
17622024-05-15T03:48:54.214ZWARNnexus (BackgroundTasks): Metric producer GC: statically disabled pending omicron#5284 background_task = metrics_producer_gc file = nexus/src/app/background/metrics_producer_gc.rs:43
17632024-05-15T03:48:54.214ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 8 reason = Timeout
17642024-05-15T03:48:54.214ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 9.49\u{b5}s iteration = 8
17652024-05-15T03:48:54.214ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 10 reason = Timeout
17662024-05-15T03:48:54.214ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 15 reason = Timeout
17672024-05-15T03:48:54.214ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
17682024-05-15T03:48:54.214ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
17692024-05-15T03:48:54.214ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
17702024-05-15T03:48:54.214ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
17712024-05-15T03:48:54.214ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
17722024-05-15T03:48:54.214ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
17732024-05-15T03:48:54.214ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
17742024-05-15T03:48:54.214ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
17752024-05-15T03:48:54.215ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
17762024-05-15T03:48:54.215ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 15 reason = Timeout
17772024-05-15T03:48:54.215ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 15 reason = Timeout
17782024-05-15T03:48:54.215ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
17792024-05-15T03:48:54.215ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
17802024-05-15T03:48:54.215ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 15 reason = Timeout
17812024-05-15T03:48:54.215ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
17822024-05-15T03:48:54.215ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
17832024-05-15T03:48:54.215ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 15 reason = Timeout
17842024-05-15T03:48:54.215ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:44
17852024-05-15T03:48:54.215ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:48
17862024-05-15T03:48:54.215ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 5.5\u{b5}s iteration = 15
17872024-05-15T03:48:54.215ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
17882024-05-15T03:48:54.215ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
17892024-05-15T03:48:54.215ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
17902024-05-15T03:48:54.215ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
17912024-05-15T03:48:54.215ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
17922024-05-15T03:48:54.216ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 15 reason = Timeout
17932024-05-15T03:48:54.216ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
17942024-05-15T03:48:54.216ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
17952024-05-15T03:48:54.216ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("3a1867b9-e5ea-4194-bfc7-0db4fb07d69e.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("9321e01e-813d-4e4f-8668-60649e9711dc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("9321e01e-813d-4e4f-8668-60649e9711dc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::8)) }], valid_until: Instant { tv_sec: 3911, tv_nsec: 640957565 } })
17962024-05-15T03:48:54.216ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("2b87d77b-ee3f-43c8-bced-dd105bbcdaa3.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("510fd65b-f60d-4faa-85be-82d26f4dd427.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("65e0e735-7f1e-479a-8127-a80c8289fcdb.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("65e0e735-7f1e-479a-8127-a80c8289fcdb.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:1::1)) }], valid_until: Instant { tv_sec: 3911, tv_nsec: 641033866 } })
17972024-05-15T03:48:54.222ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17982024-05-15T03:48:54.223ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
17992024-05-15T03:48:54.223ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 9.351585ms iteration = 8
18002024-05-15T03:48:54.223ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18012024-05-15T03:48:54.223ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18022024-05-15T03:48:54.223ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
18032024-05-15T03:48:54.223ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 9.480017ms iteration = 8
18042024-05-15T03:48:54.224ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:92
18052024-05-15T03:48:54.224ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 10.011622ms iteration = 15
18062024-05-15T03:48:54.227ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
18072024-05-15T03:48:54.227ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18082024-05-15T03:48:54.227ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3911, tv_nsec: 651645014 } })
18092024-05-15T03:48:54.227ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
18102024-05-15T03:48:54.227ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
18112024-05-15T03:48:54.227ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
18122024-05-15T03:48:54.227ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
18132024-05-15T03:48:54.228ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1022
18142024-05-15T03:48:54.228ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
18152024-05-15T03:48:54.229ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18162024-05-15T03:48:54.230ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18172024-05-15T03:48:54.231ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3911, tv_nsec: 655670205 } })
18182024-05-15T03:48:54.232ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
18192024-05-15T03:48:54.232ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
18202024-05-15T03:48:54.232ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
18212024-05-15T03:48:54.232ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1022
18222024-05-15T03:48:54.233ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
18232024-05-15T03:48:54.233ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
18242024-05-15T03:48:54.233ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
18252024-05-15T03:48:54.233ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
18262024-05-15T03:48:54.233ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
18272024-05-15T03:48:54.233ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
18282024-05-15T03:48:54.233ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18292024-05-15T03:48:54.233ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18302024-05-15T03:48:54.233ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
18312024-05-15T03:48:54.233ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18322024-05-15T03:48:54.233ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18332024-05-15T03:48:54.233ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
18342024-05-15T03:48:54.233ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
18352024-05-15T03:48:54.234ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
18362024-05-15T03:48:54.235ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
18372024-05-15T03:48:54.235ZWARNnexus (BackgroundTasks): physical disk adoption task started background_task = physical_disk_adoption file = nexus/src/app/background/physical_disk_adoption.rs:100
18382024-05-15T03:48:54.235ZWARNnexus (BackgroundTasks): Physical Disk Adoption: skipped background_task = physical_disk_adoption file = nexus/src/app/background/physical_disk_adoption.rs:104 reason = no inventory
18392024-05-15T03:48:54.235ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 20.271217ms iteration = 15
18402024-05-15T03:48:54.235ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18412024-05-15T03:48:54.235ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
18422024-05-15T03:48:54.235ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18432024-05-15T03:48:54.236ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
18442024-05-15T03:48:54.238ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18452024-05-15T03:48:54.238ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
18462024-05-15T03:48:54.238ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
18472024-05-15T03:48:54.238ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
18482024-05-15T03:48:54.239ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
18492024-05-15T03:48:54.239ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
18502024-05-15T03:48:54.239ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
18512024-05-15T03:48:54.240ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
18522024-05-15T03:48:54.241ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3911, tv_nsec: 665607286 } })
18532024-05-15T03:48:54.241ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
18542024-05-15T03:48:54.241ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
18552024-05-15T03:48:54.241ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1022 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4
18562024-05-15T03:48:54.241ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
18572024-05-15T03:48:54.242ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
18582024-05-15T03:48:54.242ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
18592024-05-15T03:48:54.242ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
18602024-05-15T03:48:54.242ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
18612024-05-15T03:48:54.242ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0b55a4be-1fdc-4b91-a3a7-f964eab4522f, lookup_type: ById(0b55a4be-1fdc-4b91-a3a7-f964eab4522f) } result = Ok(())
18622024-05-15T03:48:54.242ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
18632024-05-15T03:48:54.242ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
18642024-05-15T03:48:54.242ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
18652024-05-15T03:48:54.243ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18662024-05-15T03:48:54.244ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
18672024-05-15T03:48:54.244ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
18682024-05-15T03:48:54.244ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
18692024-05-15T03:48:54.244ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 52cc15d1-4ce1-4e38-8c6c-f9f099a8a5be dns_zone_name = oxide.test found_so_far = 0 version = 2
18702024-05-15T03:48:54.244ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
18712024-05-15T03:48:54.244ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
18722024-05-15T03:48:54.244ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
18732024-05-15T03:48:54.245ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/instance_watcher.rs:469 pruned_instances = 1 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0
18742024-05-15T03:48:54.245ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 28.906805ms iteration = 15
18752024-05-15T03:48:54.246ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18762024-05-15T03:48:54.246ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
18772024-05-15T03:48:54.246ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
18782024-05-15T03:48:54.247ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
18792024-05-15T03:48:54.247ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
18802024-05-15T03:48:54.247ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
18812024-05-15T03:48:54.248ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18822024-05-15T03:48:54.249ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
18832024-05-15T03:48:54.249ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 3698e770-e01a-43e5-856e-b62ced4fb671 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
18842024-05-15T03:48:54.249ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
18852024-05-15T03:48:54.249ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f original_time_created = 2024-05-15 03:41:43.607350 UTC target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f time_created = 2024-05-15 03:41:43.607350 UTC
18862024-05-15T03:48:54.249ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 35.763945ms iteration = 43
18872024-05-15T03:48:54.249ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
18882024-05-15T03:48:54.251ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
18892024-05-15T03:48:54.251ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
18902024-05-15T03:48:54.251ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18912024-05-15T03:48:54.251ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
18922024-05-15T03:48:54.251ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
18932024-05-15T03:48:54.251ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
18942024-05-15T03:48:54.251ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
18952024-05-15T03:48:54.251ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
18962024-05-15T03:48:54.252ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
18972024-05-15T03:48:54.253ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
18982024-05-15T03:48:54.253ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 52cc15d1-4ce1-4e38-8c6c-f9f099a8a5be dns_zone_name = oxide.test found_so_far = 1 version = 2
18992024-05-15T03:48:54.253ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
19002024-05-15T03:48:54.253ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2024-05-15 03:41:57.099854 UTC dns_group = external generation = 2
19012024-05-15T03:48:54.253ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 39.798126ms iteration = 8
19022024-05-15T03:48:54.253ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
19032024-05-15T03:48:54.255ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19042024-05-15T03:48:54.255ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
19052024-05-15T03:48:54.255ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
19062024-05-15T03:48:54.255ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
19072024-05-15T03:48:54.257ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 3698e770-e01a-43e5-856e-b62ced4fb671 dns_zone_name = control-plane.oxide.internal found_so_far = 49 version = 1
19082024-05-15T03:48:54.257ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
19092024-05-15T03:48:54.257ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2024-05-15 03:41:56.187733 UTC dns_group = internal generation = 1
19102024-05-15T03:48:54.257ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 43.570935ms iteration = 8
19112024-05-15T03:48:54.259ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 10925274-3dbb-4558-9fc7-9ec149150c20, lookup_type: ById(10925274-3dbb-4558-9fc7-9ec149150c20) } result = Ok(())
19122024-05-15T03:48:54.259ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
19132024-05-15T03:48:54.260ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
19142024-05-15T03:48:54.261ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 46.409833ms iteration = 15
19152024-05-15T03:48:54.285ZINFOnexus (BackgroundTasks): found Silo/DNS/TLS config (changed) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2024-05-15T03:41:54.396604Z, time_modified: 2024-05-15T03:41:54.396604Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 39f7e4ba-533e-4258-9f79-1873f513f101, db_silo: Silo { identity: SiloIdentity { id: 39f7e4ba-533e-4258-9f79-1873f513f101, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-05-15T03:41:57.052187Z, time_modified: 2024-05-15T03:41:57.052187Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "370e7f12febb2dc0abbebc4192727b23c734c770fdab3368ed3fd759491f6fed" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 39f7e4ba-533e-4258-9f79-1873f513f101, db_silo: Silo { identity: SiloIdentity { id: 39f7e4ba-533e-4258-9f79-1873f513f101, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-05-15T03:41:57.052187Z, time_modified: 2024-05-15T03:41:57.052187Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "370e7f12febb2dc0abbebc4192727b23c734c770fdab3368ed3fd759491f6fed" }] }) } file = nexus/src/app/background/external_endpoints.rs:101
19162024-05-15T03:48:54.285ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 70.971524ms iteration = 8
19172024-05-15T03:48:54.347ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
19182024-05-15T03:48:54.348ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
19192024-05-15T03:48:54.356ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1030 zone_address = fd00:1122:3344:101::2
19202024-05-15T03:48:54.356ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
19212024-05-15T03:48:54.358ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "7078427e-3281-49b9-b4ae-d16866a925b1", "content-length": "26", "date": "Wed, 15 May 2024 03:48:54 GMT"} })
19222024-05-15T03:48:54.358ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1035 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
19232024-05-15T03:48:54.359ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1067 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
19242024-05-15T03:48:54.374ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1030 zone_address = fd00:1122:3344:101::2
19252024-05-15T03:48:54.374ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
19262024-05-15T03:48:54.374ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "4a3f31df-4095-4075-8ce0-cd2215fb549d", "content-length": "26", "date": "Wed, 15 May 2024 03:48:54 GMT"} })
19272024-05-15T03:48:54.374ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1035 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
19282024-05-15T03:48:54.376ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1067 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
19292024-05-15T03:48:54.377ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1030 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 zone_address = fd00:1122:3344:101::2
19302024-05-15T03:48:54.377ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
19312024-05-15T03:48:54.377ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "12f86692-673a-470e-8877-d9f6271d7e95", "content-length": "26", "date": "Wed, 15 May 2024 03:48:54 GMT"} })
19322024-05-15T03:48:54.378ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1035 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
19332024-05-15T03:48:54.379ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1067 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4
19342024-05-15T03:48:54.472ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
19352024-05-15T03:48:54.473ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
19362024-05-15T03:48:54.473ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::9)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "bebaaa25-27b9-4c21-9d38-e5c1ae624aa4", "date": "Wed, 15 May 2024 03:48:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
19372024-05-15T03:48:54.474ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
19382024-05-15T03:48:54.475ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::8)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "99f45bdb-60be-4805-aed1-570d04e5a2d7", "date": "Wed, 15 May 2024 03:48:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
19392024-05-15T03:48:54.475ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
19402024-05-15T03:48:54.476ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
19412024-05-15T03:48:54.476ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 262.80644ms iteration = 10
19422024-05-15T03:48:54.582ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
19432024-05-15T03:48:54.582ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "fd80b6f3-9a21-4306-b1f7-0e2e98932eda", "content-length": "2", "date": "Wed, 15 May 2024 03:48:53 GMT"} })
19442024-05-15T03:48:54.583ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
19452024-05-15T03:48:54.584ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:3::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "58d23ccf-5b9b-41f7-b782-afe882755620", "date": "Wed, 15 May 2024 03:48:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
19462024-05-15T03:48:54.584ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
19472024-05-15T03:48:54.584ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
19482024-05-15T03:48:54.584ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
19492024-05-15T03:48:54.585ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:1::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "0431168b-f5c6-4883-949d-533fa889c62e", "date": "Wed, 15 May 2024 03:48:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
19502024-05-15T03:48:54.585ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
19512024-05-15T03:48:54.585ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 371.860393ms iteration = 15
19522024-05-15T03:48:54.586ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:2::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "b2c53a26-b7b5-4f16-b581-3051ebb7e764", "date": "Wed, 15 May 2024 03:48:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
19532024-05-15T03:48:54.586ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
19542024-05-15T03:48:54.587ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
19552024-05-15T03:48:54.587ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 373.916415ms iteration = 10
19562024-05-15T03:48:54.599ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 385.905056ms iteration = 15
19572024-05-15T03:48:54.628ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
19582024-05-15T03:48:54.628ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
19592024-05-15T03:48:54.630ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:144 port = SwitchPort { id: 89eb00e0-b12a-4307-aac5-625fd364b560, rack_id: 76c05259-5403-45cc-a1c6-20f84d7e54b4, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(7f8c724a-b675-4fd5-8a78-6b7ffc7a3247) } rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0
19602024-05-15T03:48:54.630ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
19612024-05-15T03:48:54.630ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
19622024-05-15T03:48:54.637ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
19632024-05-15T03:48:54.637ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/port/qsfp0/settings", query: Some("tag=omicron"), fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "775d4497-39bf-4e4e-b22c-a3520241e744", "content-length": "121", "date": "Wed, 15 May 2024 03:48:53 GMT"} })
19642024-05-15T03:48:54.637ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1643 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: None, kr: false, lane: Some(LinkId(0)), speed: Speed40G } }} } switch_port_id = "qsfp0"
19652024-05-15T03:48:54.637ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1670 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 7f8c724a-b675-4fd5-8a78-6b7ffc7a3247
19662024-05-15T03:48:54.637ZERROnexus (BackgroundTasks): unable to retrieve routes from switch background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1782 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0
19672024-05-15T03:48:54.637ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:369 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 routes = {}
19682024-05-15T03:48:54.637ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:373 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None)}}
19692024-05-15T03:48:54.637ZWARNnexus (BackgroundTasks): no discovered routes from switch. it is possible that an earlier api call failed. background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1511 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0
19702024-05-15T03:48:54.637ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:382 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 routes = {}
19712024-05-15T03:48:54.637ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:388 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 routes = {}
19722024-05-15T03:48:54.637ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:408 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4
19732024-05-15T03:48:54.637ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
19742024-05-15T03:48:54.637ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
19752024-05-15T03:48:54.639ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
19762024-05-15T03:48:54.639ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv4", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "2b082e03-3f4e-4346-b99b-45729663bb8c", "content-length": "2", "date": "Wed, 15 May 2024 03:48:53 GMT"} })
19772024-05-15T03:48:54.639ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
19782024-05-15T03:48:54.639ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv6", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "2b6f9b5e-c314-4b40-afd9-7607e2c8019e", "content-length": "52", "date": "Wed, 15 May 2024 03:48:53 GMT"} })
19792024-05-15T03:48:54.639ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [V4(Ipv4Network { addr: 10.151.1.174, prefix: 24 })], port: "qsfp0" }] file = nexus/src/app/background/sync_switch_configuration.rs:457 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0
19802024-05-15T03:48:54.639ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
19812024-05-15T03:48:54.688ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/switch-ports", query: None, fragment: None }, status: 204, headers: {"x-request-id": "5f86f1bc-caf2-4432-afcb-346af1baa6c5", "date": "Wed, 15 May 2024 03:48:54 GMT"} })
19822024-05-15T03:48:54.688ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:759 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch = Switch0
19832024-05-15T03:48:54.688ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
19842024-05-15T03:48:54.688ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/network-bootstore-config", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "2793c354-9844-4011-853f-241f681fa9cb", "content-length": "467", "date": "Wed, 15 May 2024 03:48:54 GMT"} })
19852024-05-15T03:48:54.688ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
19862024-05-15T03:48:54.689ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
19872024-05-15T03:48:54.691ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
19882024-05-15T03:48:54.691ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
19892024-05-15T03:48:54.694ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
19902024-05-15T03:48:54.694ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
19912024-05-15T03:48:54.696ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
19922024-05-15T03:48:54.696ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
19932024-05-15T03:48:54.702ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 486.983359ms iteration = 15
19942024-05-15T03:48:54.880ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 162 local_addr = [fd00:1122:3344:101::c]:55127 method = GET remote_addr = [fd00:1122:3344:101::d]:37120 req_id = f46a119c-864b-4fe4-93d8-f4405f8c2fa7 response_code = 200 uri = //86c56a1d-af69-43b6-8206-e5d1baefadb2
19952024-05-15T03:49:04.214ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 44 reason = Timeout
19962024-05-15T03:49:04.214ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
19972024-05-15T03:49:04.214ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
19982024-05-15T03:49:04.215ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19992024-05-15T03:49:04.218ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
20002024-05-15T03:49:04.218ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
20012024-05-15T03:49:04.218ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
20022024-05-15T03:49:04.220ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
20032024-05-15T03:49:04.220ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
20042024-05-15T03:49:04.221ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20052024-05-15T03:49:04.224ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0b55a4be-1fdc-4b91-a3a7-f964eab4522f, lookup_type: ById(0b55a4be-1fdc-4b91-a3a7-f964eab4522f) } result = Ok(())
20062024-05-15T03:49:04.224ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
20072024-05-15T03:49:04.224ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
20082024-05-15T03:49:04.230ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f original_time_created = 2024-05-15 03:41:43.607350 UTC target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f time_created = 2024-05-15 03:41:43.607350 UTC
20092024-05-15T03:49:04.230ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 16.517139ms iteration = 44
20102024-05-15T03:49:04.881ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 153 local_addr = [fd00:1122:3344:101::c]:55127 method = GET remote_addr = [fd00:1122:3344:101::d]:37120 req_id = f9cd96b7-b8d1-4a53-908d-7a0409d03903 response_code = 200 uri = //86c56a1d-af69-43b6-8206-e5d1baefadb2
20112024-05-15T03:49:14.214ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 45 reason = Timeout
20122024-05-15T03:49:14.214ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
20132024-05-15T03:49:14.214ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
20142024-05-15T03:49:14.216ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20152024-05-15T03:49:14.219ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
20162024-05-15T03:49:14.219ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
20172024-05-15T03:49:14.219ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
20182024-05-15T03:49:14.221ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
20192024-05-15T03:49:14.221ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
20202024-05-15T03:49:14.222ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20212024-05-15T03:49:14.225ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0b55a4be-1fdc-4b91-a3a7-f964eab4522f, lookup_type: ById(0b55a4be-1fdc-4b91-a3a7-f964eab4522f) } result = Ok(())
20222024-05-15T03:49:14.225ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
20232024-05-15T03:49:14.225ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
20242024-05-15T03:49:14.231ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f original_time_created = 2024-05-15 03:41:43.607350 UTC target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f time_created = 2024-05-15 03:41:43.607350 UTC
20252024-05-15T03:49:14.231ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 17.377897ms iteration = 45
20262024-05-15T03:49:14.881ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 136 local_addr = [fd00:1122:3344:101::c]:55127 method = GET remote_addr = [fd00:1122:3344:101::d]:37120 req_id = fd69fc98-7e46-4021-a5f6-a33508ab5382 response_code = 200 uri = //86c56a1d-af69-43b6-8206-e5d1baefadb2
20272024-05-15T03:49:24.214ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 46 reason = Timeout
20282024-05-15T03:49:24.215ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 16 reason = Timeout
20292024-05-15T03:49:24.215ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
20302024-05-15T03:49:24.215ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
20312024-05-15T03:49:24.215ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 16 reason = Timeout
20322024-05-15T03:49:24.215ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
20332024-05-15T03:49:24.215ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 16 reason = Timeout
20342024-05-15T03:49:24.215ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:46
20352024-05-15T03:49:24.215ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 16 reason = Timeout
20362024-05-15T03:49:24.215ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 16 reason = Timeout
20372024-05-15T03:49:24.215ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
20382024-05-15T03:49:24.215ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
20392024-05-15T03:49:24.215ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 16 reason = Timeout
20402024-05-15T03:49:24.215ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:44
20412024-05-15T03:49:24.215ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:48
20422024-05-15T03:49:24.215ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 10.57\u{b5}s iteration = 16
20432024-05-15T03:49:24.215ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 16 reason = Timeout
20442024-05-15T03:49:24.215ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
20452024-05-15T03:49:24.215ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
20462024-05-15T03:49:24.215ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
20472024-05-15T03:49:24.215ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
20482024-05-15T03:49:24.215ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
20492024-05-15T03:49:24.215ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
20502024-05-15T03:49:24.215ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
20512024-05-15T03:49:24.216ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 16 reason = Timeout
20522024-05-15T03:49:24.216ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
20532024-05-15T03:49:24.216ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
20542024-05-15T03:49:24.219ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20552024-05-15T03:49:24.221ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20562024-05-15T03:49:24.222ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3941, tv_nsec: 645761554 } })
20572024-05-15T03:49:24.223ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20582024-05-15T03:49:24.223ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
20592024-05-15T03:49:24.223ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
20602024-05-15T03:49:24.223ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
20612024-05-15T03:49:24.223ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1022
20622024-05-15T03:49:24.224ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20632024-05-15T03:49:24.224ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
20642024-05-15T03:49:24.224ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
20652024-05-15T03:49:24.224ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
20662024-05-15T03:49:24.225ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20672024-05-15T03:49:24.225ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:92
20682024-05-15T03:49:24.225ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 10.71437ms iteration = 16
20692024-05-15T03:49:24.225ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3941, tv_nsec: 649522842 } })
20702024-05-15T03:49:24.226ZWARNnexus (BackgroundTasks): physical disk adoption task started background_task = physical_disk_adoption file = nexus/src/app/background/physical_disk_adoption.rs:100
20712024-05-15T03:49:24.226ZWARNnexus (BackgroundTasks): Physical Disk Adoption: skipped background_task = physical_disk_adoption file = nexus/src/app/background/physical_disk_adoption.rs:104 reason = no inventory
20722024-05-15T03:49:24.226ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 11.406506ms iteration = 16
20732024-05-15T03:49:24.226ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1022
20742024-05-15T03:49:24.228ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
20752024-05-15T03:49:24.228ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
20762024-05-15T03:49:24.228ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
20772024-05-15T03:49:24.228ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
20782024-05-15T03:49:24.228ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
20792024-05-15T03:49:24.228ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
20802024-05-15T03:49:24.228ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
20812024-05-15T03:49:24.228ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
20822024-05-15T03:49:24.228ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
20832024-05-15T03:49:24.230ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
20842024-05-15T03:49:24.230ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
20852024-05-15T03:49:24.230ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
20862024-05-15T03:49:24.230ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
20872024-05-15T03:49:24.231ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/instance_watcher.rs:469 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0
20882024-05-15T03:49:24.231ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 15.178964ms iteration = 16
20892024-05-15T03:49:24.231ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20902024-05-15T03:49:24.232ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3941, tv_nsec: 655864557 } })
20912024-05-15T03:49:24.233ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1022 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4
20922024-05-15T03:49:24.235ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0b55a4be-1fdc-4b91-a3a7-f964eab4522f, lookup_type: ById(0b55a4be-1fdc-4b91-a3a7-f964eab4522f) } result = Ok(())
20932024-05-15T03:49:24.235ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
20942024-05-15T03:49:24.235ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
20952024-05-15T03:49:24.242ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f original_time_created = 2024-05-15 03:41:43.607350 UTC target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f time_created = 2024-05-15 03:41:43.607350 UTC
20962024-05-15T03:49:24.242ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 27.186857ms iteration = 46
20972024-05-15T03:49:24.244ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
20982024-05-15T03:49:24.244ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
20992024-05-15T03:49:24.245ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
21002024-05-15T03:49:24.246ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
21012024-05-15T03:49:24.247ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21022024-05-15T03:49:24.250ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 10925274-3dbb-4558-9fc7-9ec149150c20, lookup_type: ById(10925274-3dbb-4558-9fc7-9ec149150c20) } result = Ok(())
21032024-05-15T03:49:24.250ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
21042024-05-15T03:49:24.250ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
21052024-05-15T03:49:24.251ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 36.144489ms iteration = 16
21062024-05-15T03:49:24.309ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1030 zone_address = fd00:1122:3344:101::2
21072024-05-15T03:49:24.309ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
21082024-05-15T03:49:24.309ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "006f8ed5-3123-461a-89a3-9c12fa1c6f0d", "content-length": "26", "date": "Wed, 15 May 2024 03:49:24 GMT"} })
21092024-05-15T03:49:24.309ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1035 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
21102024-05-15T03:49:24.312ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1067 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
21112024-05-15T03:49:24.318ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1030 zone_address = fd00:1122:3344:101::2
21122024-05-15T03:49:24.318ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
21132024-05-15T03:49:24.318ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "834a3e48-5f4b-4d71-8e26-e9e8688b0841", "content-length": "26", "date": "Wed, 15 May 2024 03:49:24 GMT"} })
21142024-05-15T03:49:24.318ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1035 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
21152024-05-15T03:49:24.319ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1067 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
21162024-05-15T03:49:24.322ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1030 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 zone_address = fd00:1122:3344:101::2
21172024-05-15T03:49:24.322ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
21182024-05-15T03:49:24.322ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "a6dea2a5-a0ca-4e4b-a832-c507e40b5ced", "content-length": "26", "date": "Wed, 15 May 2024 03:49:24 GMT"} })
21192024-05-15T03:49:24.322ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1035 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
21202024-05-15T03:49:24.323ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1067 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4
21212024-05-15T03:49:24.389ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 174.579251ms iteration = 16
21222024-05-15T03:49:24.390ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
21232024-05-15T03:49:24.390ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "2c699866-df88-42c2-be70-904d33192b8c", "content-length": "2", "date": "Wed, 15 May 2024 03:49:23 GMT"} })
21242024-05-15T03:49:24.391ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
21252024-05-15T03:49:24.392ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
21262024-05-15T03:49:24.393ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 178.039636ms iteration = 16
21272024-05-15T03:49:24.450ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
21282024-05-15T03:49:24.451ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
21292024-05-15T03:49:24.452ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:144 port = SwitchPort { id: 89eb00e0-b12a-4307-aac5-625fd364b560, rack_id: 76c05259-5403-45cc-a1c6-20f84d7e54b4, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(7f8c724a-b675-4fd5-8a78-6b7ffc7a3247) } rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0
21302024-05-15T03:49:24.452ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
21312024-05-15T03:49:24.453ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
21322024-05-15T03:49:24.459ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
21332024-05-15T03:49:24.459ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/port/qsfp0/settings", query: Some("tag=omicron"), fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "8e080224-6c32-4099-a6e8-b2ef6c58b547", "content-length": "121", "date": "Wed, 15 May 2024 03:49:23 GMT"} })
21342024-05-15T03:49:24.459ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1643 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: None, kr: false, lane: Some(LinkId(0)), speed: Speed40G } }} } switch_port_id = "qsfp0"
21352024-05-15T03:49:24.459ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1670 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 7f8c724a-b675-4fd5-8a78-6b7ffc7a3247
21362024-05-15T03:49:24.459ZERROnexus (BackgroundTasks): unable to retrieve routes from switch background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1782 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0
21372024-05-15T03:49:24.459ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:369 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 routes = {}
21382024-05-15T03:49:24.459ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:373 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None)}}
21392024-05-15T03:49:24.459ZWARNnexus (BackgroundTasks): no discovered routes from switch. it is possible that an earlier api call failed. background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1511 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0
21402024-05-15T03:49:24.459ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:382 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 routes = {}
21412024-05-15T03:49:24.459ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:388 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 routes = {}
21422024-05-15T03:49:24.459ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:408 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4
21432024-05-15T03:49:24.459ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
21442024-05-15T03:49:24.460ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
21452024-05-15T03:49:24.461ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
21462024-05-15T03:49:24.461ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv4", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "97919a09-1855-44b8-9ee7-8d5aae09ddba", "content-length": "2", "date": "Wed, 15 May 2024 03:49:23 GMT"} })
21472024-05-15T03:49:24.461ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
21482024-05-15T03:49:24.461ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv6", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "7554f90d-0c0b-42ac-91f0-1740d607fb82", "content-length": "52", "date": "Wed, 15 May 2024 03:49:23 GMT"} })
21492024-05-15T03:49:24.461ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [V4(Ipv4Network { addr: 10.151.1.174, prefix: 24 })], port: "qsfp0" }] file = nexus/src/app/background/sync_switch_configuration.rs:457 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0
21502024-05-15T03:49:24.462ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
21512024-05-15T03:49:24.510ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/switch-ports", query: None, fragment: None }, status: 204, headers: {"x-request-id": "eb090415-dc9c-46bb-88b1-10ac597c350d", "date": "Wed, 15 May 2024 03:49:24 GMT"} })
21522024-05-15T03:49:24.510ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:759 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch = Switch0
21532024-05-15T03:49:24.510ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
21542024-05-15T03:49:24.514ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/network-bootstore-config", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "e206437b-cc0f-42d3-92ea-dfe34300706d", "content-length": "467", "date": "Wed, 15 May 2024 03:49:24 GMT"} })
21552024-05-15T03:49:24.514ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
21562024-05-15T03:49:24.514ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
21572024-05-15T03:49:24.516ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
21582024-05-15T03:49:24.517ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
21592024-05-15T03:49:24.519ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
21602024-05-15T03:49:24.519ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
21612024-05-15T03:49:24.520ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
21622024-05-15T03:49:24.520ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
21632024-05-15T03:49:24.525ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 310.857131ms iteration = 16
21642024-05-15T03:49:24.881ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 136 local_addr = [fd00:1122:3344:101::c]:55127 method = GET remote_addr = [fd00:1122:3344:101::d]:37120 req_id = cb760fcf-d38a-4d47-b774-1c65cd3f1f94 response_code = 200 uri = //86c56a1d-af69-43b6-8206-e5d1baefadb2
21652024-05-15T03:49:34.214ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 47 reason = Timeout
21662024-05-15T03:49:34.214ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21672024-05-15T03:49:34.215ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21682024-05-15T03:49:34.216ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21692024-05-15T03:49:34.219ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
21702024-05-15T03:49:34.219ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21712024-05-15T03:49:34.219ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21722024-05-15T03:49:34.221ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21732024-05-15T03:49:34.221ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21742024-05-15T03:49:34.222ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21752024-05-15T03:49:34.224ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0b55a4be-1fdc-4b91-a3a7-f964eab4522f, lookup_type: ById(0b55a4be-1fdc-4b91-a3a7-f964eab4522f) } result = Ok(())
21762024-05-15T03:49:34.224ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21772024-05-15T03:49:34.224ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21782024-05-15T03:49:34.230ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f original_time_created = 2024-05-15 03:41:43.607350 UTC target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f time_created = 2024-05-15 03:41:43.607350 UTC
21792024-05-15T03:49:34.230ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 15.494208ms iteration = 47
21802024-05-15T03:49:34.882ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 154 local_addr = [fd00:1122:3344:101::c]:55127 method = GET remote_addr = [fd00:1122:3344:101::d]:37120 req_id = 6bab5361-1fda-4799-a95f-0d23ed38de2f response_code = 200 uri = //86c56a1d-af69-43b6-8206-e5d1baefadb2
21812024-05-15T03:49:44.214ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 48 reason = Timeout
21822024-05-15T03:49:44.214ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21832024-05-15T03:49:44.214ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21842024-05-15T03:49:44.216ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21852024-05-15T03:49:44.219ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
21862024-05-15T03:49:44.219ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21872024-05-15T03:49:44.219ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21882024-05-15T03:49:44.223ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21892024-05-15T03:49:44.223ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21902024-05-15T03:49:44.224ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21912024-05-15T03:49:44.227ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0b55a4be-1fdc-4b91-a3a7-f964eab4522f, lookup_type: ById(0b55a4be-1fdc-4b91-a3a7-f964eab4522f) } result = Ok(())
21922024-05-15T03:49:44.227ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
21932024-05-15T03:49:44.227ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
21942024-05-15T03:49:44.233ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f original_time_created = 2024-05-15 03:41:43.607350 UTC target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f time_created = 2024-05-15 03:41:43.607350 UTC
21952024-05-15T03:49:44.233ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 18.933404ms iteration = 48
21962024-05-15T03:49:44.882ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 158 local_addr = [fd00:1122:3344:101::c]:55127 method = GET remote_addr = [fd00:1122:3344:101::d]:37120 req_id = 19b517ae-11c9-42eb-951f-b17c0500d364 response_code = 200 uri = //86c56a1d-af69-43b6-8206-e5d1baefadb2
21972024-05-15T03:49:54.215ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 17 reason = Timeout
21982024-05-15T03:49:54.215ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
21992024-05-15T03:49:54.215ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 11 reason = Timeout
22002024-05-15T03:49:54.215ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 17 reason = Timeout
22012024-05-15T03:49:54.215ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
22022024-05-15T03:49:54.215ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 17 reason = Timeout
22032024-05-15T03:49:54.215ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:44
22042024-05-15T03:49:54.215ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:48
22052024-05-15T03:49:54.215ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 7.68\u{b5}s iteration = 17
22062024-05-15T03:49:54.215ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 17 reason = Timeout
22072024-05-15T03:49:54.215ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 49 reason = Timeout
22082024-05-15T03:49:54.215ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
22092024-05-15T03:49:54.215ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
22102024-05-15T03:49:54.215ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 9 reason = Timeout
22112024-05-15T03:49:54.215ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 17 reason = Timeout
22122024-05-15T03:49:54.215ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 17 reason = Timeout
22132024-05-15T03:49:54.215ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:46
22142024-05-15T03:49:54.215ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
22152024-05-15T03:49:54.215ZDEBGnexus (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 11 reason = Timeout
22162024-05-15T03:49:54.215ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 17 reason = Timeout
22172024-05-15T03:49:54.215ZDEBGnexus (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 9 reason = Timeout
22182024-05-15T03:49:54.215ZDEBGnexus (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 9 reason = Timeout
22192024-05-15T03:49:54.215ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
22202024-05-15T03:49:54.215ZWARNnexus (BackgroundTasks): Metric producer GC: statically disabled pending omicron#5284 background_task = metrics_producer_gc file = nexus/src/app/background/metrics_producer_gc.rs:43
22212024-05-15T03:49:54.215ZDEBGnexus (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 17.74\u{b5}s iteration = 9
22222024-05-15T03:49:54.215ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_executor iteration = 10 reason = Timeout
22232024-05-15T03:49:54.215ZWARNnexus (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor file = nexus/src/app/background/blueprint_execution.rs:65 reason = blueprint disabled target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f
22242024-05-15T03:49:54.215ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 9 reason = Timeout
22252024-05-15T03:49:54.215ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
22262024-05-15T03:49:54.215ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 21.99\u{b5}s iteration = 10
22272024-05-15T03:49:54.215ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
22282024-05-15T03:49:54.215ZDEBGnexus (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 9 reason = Timeout
22292024-05-15T03:49:54.215ZDEBGnexus (BackgroundTasks): activating background_task = external_endpoints iteration = 9 reason = Timeout
22302024-05-15T03:49:54.215ZDEBGnexus (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
22312024-05-15T03:49:54.215ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
22322024-05-15T03:49:54.215ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
22332024-05-15T03:49:54.215ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
22342024-05-15T03:49:54.215ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
22352024-05-15T03:49:54.215ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
22362024-05-15T03:49:54.216ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
22372024-05-15T03:49:54.216ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
22382024-05-15T03:49:54.216ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
22392024-05-15T03:49:54.216ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
22402024-05-15T03:49:54.216ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
22412024-05-15T03:49:54.216ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
22422024-05-15T03:49:54.216ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
22432024-05-15T03:49:54.216ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 17 reason = Timeout
22442024-05-15T03:49:54.216ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
22452024-05-15T03:49:54.216ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
22462024-05-15T03:49:54.217ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
22472024-05-15T03:49:54.217ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("2b87d77b-ee3f-43c8-bced-dd105bbcdaa3.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("510fd65b-f60d-4faa-85be-82d26f4dd427.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("65e0e735-7f1e-479a-8127-a80c8289fcdb.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("65e0e735-7f1e-479a-8127-a80c8289fcdb.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:1::1)) }], valid_until: Instant { tv_sec: 3971, tv_nsec: 640327488 } })
22482024-05-15T03:49:54.217ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
22492024-05-15T03:49:54.217ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
22502024-05-15T03:49:54.217ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22512024-05-15T03:49:54.218ZDEBGnexus (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("3a1867b9-e5ea-4194-bfc7-0db4fb07d69e.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 5353, target: Name("9321e01e-813d-4e4f-8668-60649e9711dc.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("9321e01e-813d-4e4f-8668-60649e9711dc.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::8)) }], valid_until: Instant { tv_sec: 3971, tv_nsec: 641575951 } })
22522024-05-15T03:49:54.218ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22532024-05-15T03:49:54.219ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22542024-05-15T03:49:54.219ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:92
22552024-05-15T03:49:54.219ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.98947ms iteration = 17
22562024-05-15T03:49:54.219ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22572024-05-15T03:49:54.220ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22582024-05-15T03:49:54.220ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22592024-05-15T03:49:54.223ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } background_task = dns_servers_internal current_servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } dns_group = internal
22602024-05-15T03:49:54.223ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 8.395526ms iteration = 9
22612024-05-15T03:49:54.223ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
22622024-05-15T03:49:54.223ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
22632024-05-15T03:49:54.224ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
22642024-05-15T03:49:54.224ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
22652024-05-15T03:49:54.224ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
22662024-05-15T03:49:54.224ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
22672024-05-15T03:49:54.224ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
22682024-05-15T03:49:54.224ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
22692024-05-15T03:49:54.225ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
22702024-05-15T03:49:54.225ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
22712024-05-15T03:49:54.225ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
22722024-05-15T03:49:54.225ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
22732024-05-15T03:49:54.225ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
22742024-05-15T03:49:54.225ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
22752024-05-15T03:49:54.225ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
22762024-05-15T03:49:54.225ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
22772024-05-15T03:49:54.225ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
22782024-05-15T03:49:54.225ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
22792024-05-15T03:49:54.226ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22802024-05-15T03:49:54.226ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22812024-05-15T03:49:54.226ZWARNnexus (BackgroundTasks): physical disk adoption task started background_task = physical_disk_adoption file = nexus/src/app/background/physical_disk_adoption.rs:100
22822024-05-15T03:49:54.226ZWARNnexus (BackgroundTasks): Physical Disk Adoption: skipped background_task = physical_disk_adoption file = nexus/src/app/background/physical_disk_adoption.rs:104 reason = no inventory
22832024-05-15T03:49:54.226ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 10.76271ms iteration = 17
22842024-05-15T03:49:54.226ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3971, tv_nsec: 650127698 } })
22852024-05-15T03:49:54.226ZDEBGnexus (BackgroundTasks): found DNS servers (no change) addresses = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } background_task = dns_servers_external current_servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } dns_group = external
22862024-05-15T03:49:54.226ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 11.337606ms iteration = 9
22872024-05-15T03:49:54.227ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
22882024-05-15T03:49:54.227ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
22892024-05-15T03:49:54.227ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
22902024-05-15T03:49:54.227ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
22912024-05-15T03:49:54.227ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
22922024-05-15T03:49:54.227ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
22932024-05-15T03:49:54.228ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3971, tv_nsec: 651720385 } })
22942024-05-15T03:49:54.229ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22952024-05-15T03:49:54.229ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22962024-05-15T03:49:54.230ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
22972024-05-15T03:49:54.231ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
22982024-05-15T03:49:54.231ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
22992024-05-15T03:49:54.231ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
23002024-05-15T03:49:54.231ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
23012024-05-15T03:49:54.231ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
23022024-05-15T03:49:54.231ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
23032024-05-15T03:49:54.232ZDEBGnexus (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
23042024-05-15T03:49:54.232ZDEBGnexus (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
23052024-05-15T03:49:54.232ZDEBGnexus (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
23062024-05-15T03:49:54.232ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
23072024-05-15T03:49:54.232ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23082024-05-15T03:49:54.233ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
23092024-05-15T03:49:54.233ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0b55a4be-1fdc-4b91-a3a7-f964eab4522f, lookup_type: ById(0b55a4be-1fdc-4b91-a3a7-f964eab4522f) } result = Ok(())
23102024-05-15T03:49:54.233ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
23112024-05-15T03:49:54.233ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
23122024-05-15T03:49:54.234ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23132024-05-15T03:49:54.235ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
23142024-05-15T03:49:54.235ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
23152024-05-15T03:49:54.235ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1022
23162024-05-15T03:49:54.236ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
23172024-05-15T03:49:54.236ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1022
23182024-05-15T03:49:54.236ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 3971, tv_nsec: 659480394 } })
23192024-05-15T03:49:54.238ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
23202024-05-15T03:49:54.238ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
23212024-05-15T03:49:54.238ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
23222024-05-15T03:49:54.238ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
23232024-05-15T03:49:54.239ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
23242024-05-15T03:49:54.239ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
23252024-05-15T03:49:54.239ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
23262024-05-15T03:49:54.239ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
23272024-05-15T03:49:54.240ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1022 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4
23282024-05-15T03:49:54.240ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23292024-05-15T03:49:54.240ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
23302024-05-15T03:49:54.240ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 3698e770-e01a-43e5-856e-b62ced4fb671 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
23312024-05-15T03:49:54.240ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
23322024-05-15T03:49:54.240ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
23332024-05-15T03:49:54.241ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f original_time_created = 2024-05-15 03:41:43.607350 UTC target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f time_created = 2024-05-15 03:41:43.607350 UTC
23342024-05-15T03:49:54.241ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 26.404749ms iteration = 49
23352024-05-15T03:49:54.241ZDEBGnexus (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
23362024-05-15T03:49:54.241ZDEBGnexus (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 52cc15d1-4ce1-4e38-8c6c-f9f099a8a5be dns_zone_name = oxide.test found_so_far = 0 version = 2
23372024-05-15T03:49:54.242ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
23382024-05-15T03:49:54.242ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23392024-05-15T03:49:54.242ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
23402024-05-15T03:49:54.243ZDEBGnexus (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
23412024-05-15T03:49:54.243ZDEBGnexus (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
23422024-05-15T03:49:54.243ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
23432024-05-15T03:49:54.243ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23442024-05-15T03:49:54.243ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
23452024-05-15T03:49:54.244ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
23462024-05-15T03:49:54.245ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
23472024-05-15T03:49:54.246ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
23482024-05-15T03:49:54.246ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
23492024-05-15T03:49:54.246ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
23502024-05-15T03:49:54.247ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
23512024-05-15T03:49:54.247ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23522024-05-15T03:49:54.247ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
23532024-05-15T03:49:54.247ZDEBGnexus (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
23542024-05-15T03:49:54.248ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
23552024-05-15T03:49:54.249ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 3698e770-e01a-43e5-856e-b62ced4fb671 dns_zone_name = control-plane.oxide.internal found_so_far = 49 version = 1
23562024-05-15T03:49:54.249ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
23572024-05-15T03:49:54.249ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_internal current_generation = 1 current_time_created = 2024-05-15 03:41:56.187733 UTC dns_group = internal generation = 1
23582024-05-15T03:49:54.249ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 33.577793ms iteration = 9
23592024-05-15T03:49:54.249ZDEBGnexus (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 52cc15d1-4ce1-4e38-8c6c-f9f099a8a5be dns_zone_name = oxide.test found_so_far = 1 version = 2
23602024-05-15T03:49:54.249ZDEBGnexus (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
23612024-05-15T03:49:54.249ZDEBGnexus (BackgroundTasks): found latest DNS generation (unchanged) background_task = dns_config_external current_generation = 2 current_time_created = 2024-05-15 03:41:57.099854 UTC dns_group = external generation = 2
23622024-05-15T03:49:54.249ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 33.631643ms iteration = 9
23632024-05-15T03:49:54.251ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 10925274-3dbb-4558-9fc7-9ec149150c20, lookup_type: ById(10925274-3dbb-4558-9fc7-9ec149150c20) } result = Ok(())
23642024-05-15T03:49:54.251ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
23652024-05-15T03:49:54.252ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
23662024-05-15T03:49:54.257ZDEBGnexus (BackgroundTasks): found Silo/DNS/TLS config (no change) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2024-05-15T03:41:54.396604Z, time_modified: 2024-05-15T03:41:54.396604Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "recovery.sys.oxide.test": ExternalEndpoint { silo_id: 39f7e4ba-533e-4258-9f79-1873f513f101, db_silo: Silo { identity: SiloIdentity { id: 39f7e4ba-533e-4258-9f79-1873f513f101, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-05-15T03:41:57.052187Z, time_modified: 2024-05-15T03:41:57.052187Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "370e7f12febb2dc0abbebc4192727b23c734c770fdab3368ed3fd759491f6fed" }] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 39f7e4ba-533e-4258-9f79-1873f513f101, db_silo: Silo { identity: SiloIdentity { id: 39f7e4ba-533e-4258-9f79-1873f513f101, name: Name(Name("recovery")), description: "built-in recovery Silo", time_created: 2024-05-15T03:41:57.052187Z, time_modified: 2024-05-15T03:41:57.052187Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(4)) }, tls_certs: [TlsCertificate { digest: "370e7f12febb2dc0abbebc4192727b23c734c770fdab3368ed3fd759491f6fed" }] }) }
23672024-05-15T03:49:54.257ZDEBGnexus (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 42.095209ms iteration = 9
23682024-05-15T03:49:54.261ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 46.03979ms iteration = 17
23692024-05-15T03:49:54.300ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/instance_watcher.rs:469 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0
23702024-05-15T03:49:54.300ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 83.743144ms iteration = 17
23712024-05-15T03:49:54.326ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:2::1]:5353/config
23722024-05-15T03:49:54.343ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::9]:5353/config
23732024-05-15T03:49:54.376ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1030 zone_address = fd00:1122:3344:101::2
23742024-05-15T03:49:54.376ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
23752024-05-15T03:49:54.377ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "3a30d443-0069-4989-9aa5-c351104ec645", "content-length": "26", "date": "Wed, 15 May 2024 03:49:54 GMT"} })
23762024-05-15T03:49:54.378ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1035 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
23772024-05-15T03:49:54.379ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1067 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
23782024-05-15T03:49:54.381ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1030 zone_address = fd00:1122:3344:101::2
23792024-05-15T03:49:54.381ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
23802024-05-15T03:49:54.382ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "b8b8c37b-c174-4852-92bf-d37e6b451fef", "content-length": "26", "date": "Wed, 15 May 2024 03:49:54 GMT"} })
23812024-05-15T03:49:54.382ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1035 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
23822024-05-15T03:49:54.384ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1067 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
23832024-05-15T03:49:54.401ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1030 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 zone_address = fd00:1122:3344:101::2
23842024-05-15T03:49:54.401ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
23852024-05-15T03:49:54.402ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "602595de-4f5e-425e-966c-f87c6c3b4803", "content-length": "26", "date": "Wed, 15 May 2024 03:49:54 GMT"} })
23862024-05-15T03:49:54.402ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1035 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
23872024-05-15T03:49:54.403ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1067 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4
23882024-05-15T03:49:54.505ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] } uri = http://[fd00:1122:3344:101::8]:5353/config
23892024-05-15T03:49:54.505ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::8)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "88dc1f3e-d304-4cd2-9bcb-a21efdb69035", "date": "Wed, 15 May 2024 03:49:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
23902024-05-15T03:49:54.505ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::8]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
23912024-05-15T03:49:54.506ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 generation = 2 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::9)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "66b5e57c-6c91-4b6c-9e53-385728dfb9a7", "date": "Wed, 15 May 2024 03:49:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
23922024-05-15T03:49:54.506ZINFOnexus (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[fd00:1122:3344:101::9]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
23932024-05-15T03:49:54.507ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external file = nexus/src/app/background/dns_propagation.rs:106 generation = 2 servers = DnsServersList { addresses: [[fd00:1122:3344:101::9]:5353, [fd00:1122:3344:101::8]:5353] }
23942024-05-15T03:49:54.507ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 292.470924ms iteration = 11
23952024-05-15T03:49:54.512ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:3::1]:5353/config
23962024-05-15T03:49:54.517ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
23972024-05-15T03:49:54.517ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "d20a4b7b-8314-4c54-9cef-b15f3b166b32", "content-length": "2", "date": "Wed, 15 May 2024 03:49:53 GMT"} })
23982024-05-15T03:49:54.519ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
23992024-05-15T03:49:54.519ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
24002024-05-15T03:49:54.520ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 305.143442ms iteration = 17
24012024-05-15T03:49:54.527ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 311.509318ms iteration = 17
24022024-05-15T03:49:54.639ZDEBGnexus (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 method = PUT servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] } uri = http://[fd00:1122:3344:1::1]:5353/config
24032024-05-15T03:49:54.639ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:2::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "78b5a28e-eb80-4967-9dc0-8fc443da4826", "date": "Wed, 15 May 2024 03:49:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
24042024-05-15T03:49:54.639ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:2::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
24052024-05-15T03:49:54.641ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:1::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "7a2c1a32-edef-4d4a-a4a9-6f94b5fda004", "date": "Wed, 15 May 2024 03:49:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
24062024-05-15T03:49:54.641ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:1::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
24072024-05-15T03:49:54.644ZDEBGnexus (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 generation = 1 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:3::1)), port: Some(5353), path: "/config", query: None, fragment: None }, status: 204, headers: {"x-request-id": "d317442b-424b-4c5f-9efb-aa351a029c85", "date": "Wed, 15 May 2024 03:49:54 GMT"} }) servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
24082024-05-15T03:49:54.644ZINFOnexus (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[fd00:1122:3344:3::1]:5353 file = nexus/src/app/background/dns_propagation.rs:169 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
24092024-05-15T03:49:54.646ZINFOnexus (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal file = nexus/src/app/background/dns_propagation.rs:106 generation = 1 servers = DnsServersList { addresses: [[fd00:1122:3344:2::1]:5353, [fd00:1122:3344:3::1]:5353, [fd00:1122:3344:1::1]:5353] }
24102024-05-15T03:49:54.646ZDEBGnexus (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 430.674123ms iteration = 11
24112024-05-15T03:49:54.689ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24122024-05-15T03:49:54.689ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24132024-05-15T03:49:54.690ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:144 port = SwitchPort { id: 89eb00e0-b12a-4307-aac5-625fd364b560, rack_id: 76c05259-5403-45cc-a1c6-20f84d7e54b4, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(7f8c724a-b675-4fd5-8a78-6b7ffc7a3247) } rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0
24142024-05-15T03:49:54.690ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24152024-05-15T03:49:54.691ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24162024-05-15T03:49:54.700ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
24172024-05-15T03:49:54.700ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/port/qsfp0/settings", query: Some("tag=omicron"), fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "f5365bac-be94-45e6-9027-372e79fe70e9", "content-length": "121", "date": "Wed, 15 May 2024 03:49:53 GMT"} })
24182024-05-15T03:49:54.700ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1643 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: None, kr: false, lane: Some(LinkId(0)), speed: Speed40G } }} } switch_port_id = "qsfp0"
24192024-05-15T03:49:54.700ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1670 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 7f8c724a-b675-4fd5-8a78-6b7ffc7a3247
24202024-05-15T03:49:54.700ZERROnexus (BackgroundTasks): unable to retrieve routes from switch background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1782 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0
24212024-05-15T03:49:54.700ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:369 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 routes = {}
24222024-05-15T03:49:54.700ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:373 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None)}}
24232024-05-15T03:49:54.700ZWARNnexus (BackgroundTasks): no discovered routes from switch. it is possible that an earlier api call failed. background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1511 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0
24242024-05-15T03:49:54.700ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:382 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 routes = {}
24252024-05-15T03:49:54.700ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:388 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 routes = {}
24262024-05-15T03:49:54.700ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:408 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4
24272024-05-15T03:49:54.700ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24282024-05-15T03:49:54.700ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24292024-05-15T03:49:54.702ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
24302024-05-15T03:49:54.702ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv4", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "b3103b23-3ac8-4939-984a-fd343ee77afa", "content-length": "2", "date": "Wed, 15 May 2024 03:49:53 GMT"} })
24312024-05-15T03:49:54.702ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
24322024-05-15T03:49:54.702ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv6", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "830fbbfa-ed18-4b6f-953f-defff7e66bda", "content-length": "52", "date": "Wed, 15 May 2024 03:49:53 GMT"} })
24332024-05-15T03:49:54.702ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [V4(Ipv4Network { addr: 10.151.1.174, prefix: 24 })], port: "qsfp0" }] file = nexus/src/app/background/sync_switch_configuration.rs:457 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0
24342024-05-15T03:49:54.702ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
24352024-05-15T03:49:54.768ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/switch-ports", query: None, fragment: None }, status: 204, headers: {"x-request-id": "fe2606a2-aedb-42de-a431-0b3267530665", "date": "Wed, 15 May 2024 03:49:54 GMT"} })
24362024-05-15T03:49:54.768ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:759 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch = Switch0
24372024-05-15T03:49:54.768ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
24382024-05-15T03:49:54.769ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/network-bootstore-config", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "670928c1-07c0-44a4-9033-999191c4e15b", "content-length": "467", "date": "Wed, 15 May 2024 03:49:54 GMT"} })
24392024-05-15T03:49:54.769ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24402024-05-15T03:49:54.769ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24412024-05-15T03:49:54.771ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24422024-05-15T03:49:54.772ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24432024-05-15T03:49:54.774ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24442024-05-15T03:49:54.774ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24452024-05-15T03:49:54.775ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24462024-05-15T03:49:54.775ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
24472024-05-15T03:49:54.781ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 565.897063ms iteration = 17
24482024-05-15T03:49:54.882ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 173 local_addr = [fd00:1122:3344:101::c]:55127 method = GET remote_addr = [fd00:1122:3344:101::d]:37120 req_id = 43193d9b-b964-4abf-a46c-e8a753de9054 response_code = 200 uri = //86c56a1d-af69-43b6-8206-e5d1baefadb2
24492024-05-15T03:50:04.214ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 50 reason = Timeout
24502024-05-15T03:50:04.214ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24512024-05-15T03:50:04.215ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24522024-05-15T03:50:04.217ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24532024-05-15T03:50:04.219ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
24542024-05-15T03:50:04.219ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24552024-05-15T03:50:04.219ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24562024-05-15T03:50:04.221ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24572024-05-15T03:50:04.221ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24582024-05-15T03:50:04.222ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24592024-05-15T03:50:04.225ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0b55a4be-1fdc-4b91-a3a7-f964eab4522f, lookup_type: ById(0b55a4be-1fdc-4b91-a3a7-f964eab4522f) } result = Ok(())
24602024-05-15T03:50:04.225ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24612024-05-15T03:50:04.225ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24622024-05-15T03:50:04.231ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f original_time_created = 2024-05-15 03:41:43.607350 UTC target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f time_created = 2024-05-15 03:41:43.607350 UTC
24632024-05-15T03:50:04.231ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 16.953063ms iteration = 50
24642024-05-15T03:50:04.881ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 172 local_addr = [fd00:1122:3344:101::c]:55127 method = GET remote_addr = [fd00:1122:3344:101::d]:37120 req_id = bfe0511c-0af3-4f1d-be7d-095a9a3fbf1c response_code = 200 uri = //86c56a1d-af69-43b6-8206-e5d1baefadb2
24652024-05-15T03:50:14.215ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 51 reason = Timeout
24662024-05-15T03:50:14.215ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24672024-05-15T03:50:14.215ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24682024-05-15T03:50:14.217ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24692024-05-15T03:50:14.220ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
24702024-05-15T03:50:14.220ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24712024-05-15T03:50:14.220ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24722024-05-15T03:50:14.225ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24732024-05-15T03:50:14.226ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24742024-05-15T03:50:14.227ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24752024-05-15T03:50:14.229ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0b55a4be-1fdc-4b91-a3a7-f964eab4522f, lookup_type: ById(0b55a4be-1fdc-4b91-a3a7-f964eab4522f) } result = Ok(())
24762024-05-15T03:50:14.229ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24772024-05-15T03:50:14.229ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
24782024-05-15T03:50:14.235ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f original_time_created = 2024-05-15 03:41:43.607350 UTC target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f time_created = 2024-05-15 03:41:43.607350 UTC
24792024-05-15T03:50:14.235ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.60592ms iteration = 51
24802024-05-15T03:50:14.882ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 141 local_addr = [fd00:1122:3344:101::c]:55127 method = GET remote_addr = [fd00:1122:3344:101::d]:37120 req_id = ed61c4d9-39ec-4d08-9bc1-af945361b6bd response_code = 200 uri = //86c56a1d-af69-43b6-8206-e5d1baefadb2
24812024-05-15T03:50:24.215ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 52 reason = Timeout
24822024-05-15T03:50:24.215ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
24832024-05-15T03:50:24.215ZDEBGnexus (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 18 reason = Timeout
24842024-05-15T03:50:24.215ZDEBGnexus (BackgroundTasks): activating background_task = bfd_manager iteration = 18 reason = Timeout
24852024-05-15T03:50:24.215ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
24862024-05-15T03:50:24.215ZDEBGnexus (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
24872024-05-15T03:50:24.215ZDEBGnexus (BackgroundTasks): activating background_task = phantom_disks iteration = 18 reason = Timeout
24882024-05-15T03:50:24.215ZWARNnexus (BackgroundTasks): phantom disk task started background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:46
24892024-05-15T03:50:24.215ZDEBGnexus (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 18 reason = Timeout
24902024-05-15T03:50:24.215ZDEBGnexus (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 18 reason = Timeout
24912024-05-15T03:50:24.215ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
24922024-05-15T03:50:24.215ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
24932024-05-15T03:50:24.215ZDEBGnexus (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 18 reason = Timeout
24942024-05-15T03:50:24.215ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
24952024-05-15T03:50:24.216ZDEBGnexus (BackgroundTasks): activating background_task = region_replacement iteration = 18 reason = Timeout
24962024-05-15T03:50:24.216ZWARNnexus (BackgroundTasks): region replacement task started background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:44
24972024-05-15T03:50:24.216ZWARNnexus (BackgroundTasks): region replacement task done background_task = region_replacement file = nexus/src/app/background/region_replacement.rs:48
24982024-05-15T03:50:24.216ZDEBGnexus (BackgroundTasks): activation complete background_task = region_replacement elapsed = 18.451\u{b5}s iteration = 18
24992024-05-15T03:50:24.216ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25002024-05-15T03:50:24.216ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
25012024-05-15T03:50:24.216ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
25022024-05-15T03:50:24.216ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
25032024-05-15T03:50:24.216ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
25042024-05-15T03:50:24.216ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
25052024-05-15T03:50:24.216ZDEBGnexus (BackgroundTasks): activating background_task = instance_watcher iteration = 18 reason = Timeout
25062024-05-15T03:50:24.216ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
25072024-05-15T03:50:24.217ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
25082024-05-15T03:50:24.218ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25092024-05-15T03:50:24.218ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25102024-05-15T03:50:24.218ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25112024-05-15T03:50:24.218ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25122024-05-15T03:50:24.219ZWARNnexus (BackgroundTasks): phantom disk task done background_task = phantom_disks file = nexus/src/app/background/phantom_disks.rs:92
25132024-05-15T03:50:24.219ZDEBGnexus (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 3.738228ms iteration = 18
25142024-05-15T03:50:24.219ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25152024-05-15T03:50:24.220ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 4001, tv_nsec: 643263478 } })
25162024-05-15T03:50:24.221ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager file = nexus/src/app/mod.rs:1022
25172024-05-15T03:50:24.222ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Fleet result = Ok(())
25182024-05-15T03:50:24.222ZDEBGnexus (BackgroundTasks): roles background_task = physical_disk_adoption roles = RoleSet { roles: {} }
25192024-05-15T03:50:24.222ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = physical_disk_adoption resource = Database result = Ok(())
25202024-05-15T03:50:24.222ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 4001, tv_nsec: 645744564 } })
25212024-05-15T03:50:24.223ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
25222024-05-15T03:50:24.223ZDEBGnexus (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
25232024-05-15T03:50:24.223ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
25242024-05-15T03:50:24.223ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
25252024-05-15T03:50:24.223ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
25262024-05-15T03:50:24.223ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
25272024-05-15T03:50:24.223ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
25282024-05-15T03:50:24.223ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
25292024-05-15T03:50:24.223ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25302024-05-15T03:50:24.223ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
25312024-05-15T03:50:24.223ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25322024-05-15T03:50:24.223ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
25332024-05-15T03:50:24.224ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1022
25342024-05-15T03:50:24.224ZWARNnexus (BackgroundTasks): physical disk adoption task started background_task = physical_disk_adoption file = nexus/src/app/background/physical_disk_adoption.rs:100
25352024-05-15T03:50:24.224ZWARNnexus (BackgroundTasks): Physical Disk Adoption: skipped background_task = physical_disk_adoption file = nexus/src/app/background/physical_disk_adoption.rs:104 reason = no inventory
25362024-05-15T03:50:24.224ZDEBGnexus (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 8.940952ms iteration = 18
25372024-05-15T03:50:24.225ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
25382024-05-15T03:50:24.225ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
25392024-05-15T03:50:24.225ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25402024-05-15T03:50:24.225ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25412024-05-15T03:50:24.227ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25422024-05-15T03:50:24.227ZDEBGnexus (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12224, target: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-10925274-3dbb-4558-9fc7-9ec149150c20.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(fd00:1122:3344:101::2)) }], valid_until: Instant { tv_sec: 4001, tv_nsec: 650006727 } })
25432024-05-15T03:50:24.228ZINFOnexus (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1022 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4
25442024-05-15T03:50:24.228ZINFOnexus (BackgroundTasks): all instance checks complete background_task = instance_watcher file = nexus/src/app/background/instance_watcher.rs:469 pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0
25452024-05-15T03:50:24.228ZDEBGnexus (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 11.652589ms iteration = 18
25462024-05-15T03:50:24.229ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0b55a4be-1fdc-4b91-a3a7-f964eab4522f, lookup_type: ById(0b55a4be-1fdc-4b91-a3a7-f964eab4522f) } result = Ok(())
25472024-05-15T03:50:24.229ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
25482024-05-15T03:50:24.230ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
25492024-05-15T03:50:24.236ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f original_time_created = 2024-05-15 03:41:43.607350 UTC target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f time_created = 2024-05-15 03:41:43.607350 UTC
25502024-05-15T03:50:24.236ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 20.312718ms iteration = 52
25512024-05-15T03:50:24.239ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
25522024-05-15T03:50:24.239ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
25532024-05-15T03:50:24.240ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
25542024-05-15T03:50:24.240ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
25552024-05-15T03:50:24.241ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25562024-05-15T03:50:24.244ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 10925274-3dbb-4558-9fc7-9ec149150c20, lookup_type: ById(10925274-3dbb-4558-9fc7-9ec149150c20) } result = Ok(())
25572024-05-15T03:50:24.244ZDEBGnexus (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
25582024-05-15T03:50:24.244ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
25592024-05-15T03:50:24.245ZDEBGnexus (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 29.621562ms iteration = 18
25602024-05-15T03:50:24.299ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1030 zone_address = fd00:1122:3344:101::2
25612024-05-15T03:50:24.299ZDEBGnexus (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
25622024-05-15T03:50:24.300ZDEBGnexus (MgsClient): client response background_task = bfd_manager result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "9e8841ce-abab-48df-a316-6768396cdc05", "content-length": "26", "date": "Wed, 15 May 2024 03:50:24 GMT"} })
25632024-05-15T03:50:24.300ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = bfd_manager file = nexus/src/app/mod.rs:1035 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
25642024-05-15T03:50:24.301ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = bfd_manager file = nexus/src/app/mod.rs:1067 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
25652024-05-15T03:50:24.304ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1030 zone_address = fd00:1122:3344:101::2
25662024-05-15T03:50:24.304ZDEBGnexus (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
25672024-05-15T03:50:24.304ZDEBGnexus (MgsClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "420b24ea-36cd-42d1-9fbc-14e4338221e4", "content-length": "26", "date": "Wed, 15 May 2024 03:50:24 GMT"} })
25682024-05-15T03:50:24.304ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1035 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
25692024-05-15T03:50:24.306ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = nat_v4_garbage_collector file = nexus/src/app/mod.rs:1067 mappings = {\n Switch0: fd00:1122:3344:101::2,\n}
25702024-05-15T03:50:24.308ZINFOnexus (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1030 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 zone_address = fd00:1122:3344:101::2
25712024-05-15T03:50:24.308ZDEBGnexus (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::2]:12225/local/switch-id
25722024-05-15T03:50:24.308ZDEBGnexus (MgsClient): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12225), path: "/local/switch-id", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "8e3c15bc-aa43-4778-be9f-e06f1a87836b", "content-length": "26", "date": "Wed, 15 May 2024 03:50:24 GMT"} })
25732024-05-15T03:50:24.308ZINFOnexus (BackgroundTasks): identified switch slot for dendrite zone background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1035 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 slot = SpIdentifier {\n slot: 0,\n type_: Switch,\n} zone_address = fd00:1122:3344:101::2
25742024-05-15T03:50:24.309ZINFOnexus (BackgroundTasks): completed mapping dendrite zones to switch slots background_task = switch_port_config_manager file = nexus/src/app/mod.rs:1067 mappings = {\n Switch0: fd00:1122:3344:101::2,\n} rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4
25752024-05-15T03:50:24.378ZDEBGnexus (DpdClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[fd00:1122:3344:101::2]:12224/rpw/nat/ipv4/gen
25762024-05-15T03:50:24.378ZDEBGnexus (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 162.78002ms iteration = 18
25772024-05-15T03:50:24.379ZDEBGnexus (DpdClient): client response background_task = nat_v4_garbage_collector result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/rpw/nat/ipv4/gen", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "22518a07-bd01-4ab7-88e3-7f1d71ef6a73", "content-length": "2", "date": "Wed, 15 May 2024 03:50:24 GMT"} })
25782024-05-15T03:50:24.380ZDEBGnexus (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
25792024-05-15T03:50:24.381ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
25802024-05-15T03:50:24.382ZDEBGnexus (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 166.628809ms iteration = 18
25812024-05-15T03:50:24.439ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
25822024-05-15T03:50:24.439ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
25832024-05-15T03:50:24.441ZINFOnexus (BackgroundTasks): fetching switch port settings background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:144 port = SwitchPort { id: 89eb00e0-b12a-4307-aac5-625fd364b560, rack_id: 76c05259-5403-45cc-a1c6-20f84d7e54b4, switch_location: "switch0", port_name: "qsfp0", port_settings_id: Some(7f8c724a-b675-4fd5-8a78-6b7ffc7a3247) } rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0
25842024-05-15T03:50:24.441ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
25852024-05-15T03:50:24.441ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
25862024-05-15T03:50:24.450ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::2]:12224/port/qsfp0/settings?tag=omicron
25872024-05-15T03:50:24.450ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/port/qsfp0/settings", query: Some("tag=omicron"), fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "cae5fa8f-5838-4bdd-aea5-1617a77e3feb", "content-length": "121", "date": "Wed, 15 May 2024 03:50:24 GMT"} })
25882024-05-15T03:50:24.450ZINFOnexus (BackgroundTasks): retrieved port settings from switch background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1643 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 settings = PortSettings { links: {"0": LinkSettings { addrs: [10.151.1.174], params: LinkCreate { autoneg: false, fec: None, kr: false, lane: Some(LinkId(0)), speed: Speed40G } }} } switch_port_id = "qsfp0"
25892024-05-15T03:50:24.450ZINFOnexus (BackgroundTasks): port settings up to date, skipping background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1670 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0 switch_port_id = "qsfp0" switch_port_settings_id = 7f8c724a-b675-4fd5-8a78-6b7ffc7a3247
25902024-05-15T03:50:24.451ZERROnexus (BackgroundTasks): unable to retrieve routes from switch background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1782 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0
25912024-05-15T03:50:24.451ZINFOnexus (BackgroundTasks): retrieved existing routes background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:369 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 routes = {}
25922024-05-15T03:50:24.451ZINFOnexus (BackgroundTasks): retrieved desired routes background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:373 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 routes = {Switch0: {(10.151.1.100, Prefix4 { length: 0, value: 0.0.0.0 }, None)}}
25932024-05-15T03:50:24.451ZWARNnexus (BackgroundTasks): no discovered routes from switch. it is possible that an earlier api call failed. background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:1511 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0
25942024-05-15T03:50:24.451ZINFOnexus (BackgroundTasks): calculated static routes to add background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:382 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 routes = {}
25952024-05-15T03:50:24.451ZINFOnexus (BackgroundTasks): calculated static routes to delete background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:388 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 routes = {}
25962024-05-15T03:50:24.451ZINFOnexus (BackgroundTasks): checking for changes to loopback addresses background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:408 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4
25972024-05-15T03:50:24.451ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
25982024-05-15T03:50:24.451ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
25992024-05-15T03:50:24.452ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv4
26002024-05-15T03:50:24.452ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv4", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "697e74e3-d744-4e06-a9e3-5b46d5cab060", "content-length": "2", "date": "Wed, 15 May 2024 03:50:24 GMT"} })
26012024-05-15T03:50:24.452ZDEBGnexus (DpdClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::2]:12224/loopback/ipv6
26022024-05-15T03:50:24.453ZDEBGnexus (DpdClient): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::2)), port: Some(12224), path: "/loopback/ipv6", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "f8def7da-ff19-4c96-a94f-985148dbaff7", "content-length": "52", "date": "Wed, 15 May 2024 03:50:24 GMT"} })
26032024-05-15T03:50:24.453ZINFOnexus (BackgroundTasks): applying SMF config uplink updates to switch zone background_task = switch_port_config_manager config = [HostPortConfig { addrs: [V4(Ipv4Network { addr: 10.151.1.174, prefix: 24 })], port: "qsfp0" }] file = nexus/src/app/background/sync_switch_configuration.rs:457 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch_location = Switch0
26042024-05-15T03:50:24.453ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = Some(Body) method = POST rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::1]:12345/switch-ports
26052024-05-15T03:50:24.501ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/switch-ports", query: None, fragment: None }, status: 204, headers: {"x-request-id": "d6eedd13-6cd7-4947-9f2b-7d13be1d44ef", "date": "Wed, 15 May 2024 03:50:24 GMT"} })
26062024-05-15T03:50:24.502ZINFOnexus (BackgroundTasks): no bgp config found for switch, skipping. background_task = switch_port_config_manager file = nexus/src/app/background/sync_switch_configuration.rs:759 rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 switch = Switch0
26072024-05-15T03:50:24.502ZDEBGnexus (BackgroundTasks): client request background_task = switch_port_config_manager body = None method = GET rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 uri = http://[fd00:1122:3344:101::1]:12345/network-bootstore-config
26082024-05-15T03:50:24.505ZDEBGnexus (BackgroundTasks): client response background_task = switch_port_config_manager rack_id = 76c05259-5403-45cc-a1c6-20f84d7e54b4 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::1)), port: Some(12345), path: "/network-bootstore-config", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "b03df77d-9a4b-41d6-b1ad-a89ef2dde541", "content-length": "467", "date": "Wed, 15 May 2024 03:50:24 GMT"} })
26092024-05-15T03:50:24.505ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
26102024-05-15T03:50:24.505ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
26112024-05-15T03:50:24.508ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
26122024-05-15T03:50:24.508ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
26132024-05-15T03:50:24.510ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
26142024-05-15T03:50:24.510ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
26152024-05-15T03:50:24.511ZDEBGnexus (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
26162024-05-15T03:50:24.512ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
26172024-05-15T03:50:24.517ZDEBGnexus (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 301.320204ms iteration = 18
26182024-05-15T03:50:24.882ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 221 local_addr = [fd00:1122:3344:101::c]:55127 method = GET remote_addr = [fd00:1122:3344:101::d]:37120 req_id = 67b979f3-88ce-45d0-ae1d-cf49eb5f1cf1 response_code = 200 uri = //86c56a1d-af69-43b6-8206-e5d1baefadb2
26192024-05-15T03:50:24.884ZDEBGnexus (producer-registration-task): registering / renewing oximeter producer lease with Nexus
26202024-05-15T03:50:24.884ZDEBGnexus (producer-registration-task): using nexus address for registration addr = [fd00:1122:3344:101::c]:12221
26212024-05-15T03:50:24.914ZDEBGnexus (producer-registration-task): client request body = Some(Body) method = POST uri = http://[fd00:1122:3344:101::c]:12221/metrics/producers
26222024-05-15T03:50:24.914ZINFOnexus (dropshot_internal): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::c]:12221 remote_addr = [fd00:1122:3344:101::c]:48778
26232024-05-15T03:50:24.914ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::c]:48778 req_id = 0f4f57ff-7ee6-49c0-928d-71ebbec9ccb7 roles = RoleSet { roles: {} } uri = /metrics/producers
26242024-05-15T03:50:24.914ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::c]:48778 req_id = 0f4f57ff-7ee6-49c0-928d-71ebbec9ccb7 resource = Database result = Ok(()) uri = /metrics/producers
26252024-05-15T03:50:24.946ZINFOnexus: registered oximeter collector client file = nexus/src/app/oximeter.rs:338 id = f12aa7e2-1873-452c-b215-b2593f228861
26262024-05-15T03:50:24.946ZDEBGnexus (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::c]:48778 req_id = 0f4f57ff-7ee6-49c0-928d-71ebbec9ccb7 roles = RoleSet { roles: {} } uri = /metrics/producers
26272024-05-15T03:50:24.946ZDEBGnexus (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::c]:48778 req_id = 0f4f57ff-7ee6-49c0-928d-71ebbec9ccb7 resource = Database result = Ok(()) uri = /metrics/producers
26282024-05-15T03:50:24.981ZDEBGnexus: client request body = Some(Body) method = POST oximeter-collector = f12aa7e2-1873-452c-b215-b2593f228861 uri = http://[fd00:1122:3344:101::d]:12223/producers
26292024-05-15T03:50:24.981ZDEBGnexus: client response oximeter-collector = f12aa7e2-1873-452c-b215-b2593f228861 result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::d)), port: Some(12223), path: "/producers", query: None, fragment: None }, status: 204, headers: {"x-request-id": "5dbf70e4-0b44-447c-bfbf-0ab83c9fab40", "date": "Wed, 15 May 2024 03:50:24 GMT"} })
26302024-05-15T03:50:24.981ZINFOnexus: assigned collector to new producer collector_id = f12aa7e2-1873-452c-b215-b2593f228861 file = nexus/src/app/oximeter.rs:125 producer_id = 86c56a1d-af69-43b6-8206-e5d1baefadb2
26312024-05-15T03:50:24.982ZINFOnexus (dropshot_internal): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 68658 local_addr = [fd00:1122:3344:101::c]:12221 method = POST remote_addr = [fd00:1122:3344:101::c]:48778 req_id = 0f4f57ff-7ee6-49c0-928d-71ebbec9ccb7 response_code = 201 uri = /metrics/producers
26322024-05-15T03:50:24.982ZDEBGnexus (producer-registration-task): client response result = Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(fd00:1122:3344:101::c)), port: Some(12221), path: "/metrics/producers", query: None, fragment: None }, status: 201, headers: {"content-type": "application/json", "x-request-id": "0f4f57ff-7ee6-49c0-928d-71ebbec9ccb7", "content-length": "41", "date": "Wed, 15 May 2024 03:50:24 GMT"} })
26332024-05-15T03:50:24.983ZDEBGnexus (producer-registration-task): registered with nexus successfully
26342024-05-15T03:50:24.983ZDEBGnexus (producer-registration-task): pausing until time to renew lease lease_duration = 600s wait_period = 150s
26352024-05-15T03:50:34.215ZDEBGnexus (BackgroundTasks): activating background_task = blueprint_loader iteration = 53 reason = Timeout
26362024-05-15T03:50:34.215ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
26372024-05-15T03:50:34.215ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
26382024-05-15T03:50:34.217ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26392024-05-15T03:50:34.220ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
26402024-05-15T03:50:34.220ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
26412024-05-15T03:50:34.220ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
26422024-05-15T03:50:34.221ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
26432024-05-15T03:50:34.221ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
26442024-05-15T03:50:34.223ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26452024-05-15T03:50:34.225ZDEBGnexus (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Blueprint { parent: Fleet, key: 0b55a4be-1fdc-4b91-a3a7-f964eab4522f, lookup_type: ById(0b55a4be-1fdc-4b91-a3a7-f964eab4522f) } result = Ok(())
26462024-05-15T03:50:34.225ZDEBGnexus (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
26472024-05-15T03:50:34.225ZDEBGnexus (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
26482024-05-15T03:50:34.231ZDEBGnexus (BackgroundTasks): found latest target blueprint (unchanged) background_task = blueprint_loader original_target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f original_time_created = 2024-05-15 03:41:43.607350 UTC target_id = 0b55a4be-1fdc-4b91-a3a7-f964eab4522f time_created = 2024-05-15 03:41:43.607350 UTC
26492024-05-15T03:50:34.231ZDEBGnexus (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 16.64198ms iteration = 53
26502024-05-15T03:50:34.983ZINFOnexus (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 1003 local_addr = [fd00:1122:3344:101::c]:55127 method = GET remote_addr = [fd00:1122:3344:101::d]:37120 req_id = 9b1e7b83-dd15-4039-b331-6bde098b35ae response_code = 200 uri = //86c56a1d-af69-43b6-8206-e5d1baefadb2